meilisearch @ v1.40.0

This release introduced support for the distinct attribute in federated search, enabling cross-index distinct attributes with facet distribution support. Additionally, significant performance improvements were delivered, including faster federated search (approximately 100ms faster), optimized JSON document generation for better handling of large documents and a much better memory usage for large workloads.

✨ Enhancement

• Support distinct in federated search by @dureuill in https://github.com/meilisearch/meilisearch/pull/6214

  The distinct attribute can now be passed to the federation object in federated search to apply a global, cross-index and cross-remote distinct computation to the results.

  {
    "federation": {
      "distinct": "genres", // ✨ NEW
      "facetsByIndex": { // recovering facet distribution is also supported with distinct
        "comics": [
          "genres"
        ],
        "movies": [
          "genres"
        ]
      },
      "mergeFacets": {} // merging facet distributions is also supported with distinct
    },
    "queries": [
      {
        "indexUid": "comics",
        "q": "batman",
        "attributesToRetrieve": ["title", "genres"],
        "useNetwork": true // distinct is also supported with network queries
      },
      {
        "indexUid": "movies",
        "q": "superman",
        "attributesToRetrieve": ["title", "genres"],
        "useNetwork": true
      }
    ]
  }
  

  {
    "hits": [
      {
        "title": "Batman",
        "genres": [
          "Family",
          "Adventure",
          "Comedy",
          "Science Fiction",
          "Crime"
        ],
        "_federation": {
          "indexUid": "comics",
          "queriesPosition": 0,
          "weightedRankingScore": 1.0,
          "remote": "ms2"
        }
      },
      {
        "title": "Batman",
        "genres": [
          "Fantasy",
          "Action"
        ],
        "_federation": {
          "indexUid": "comics",
          "queriesPosition": 0,
          "weightedRankingScore": 1.0,
          "remote": "ms1"
        }
      },
      {
        "title": "Batman & Bill",
        "genres": [
          "Documentary"
        ],
        "_federation": {
          "indexUid": "comics",
          "queriesPosition": 0,
          "weightedRankingScore": 0.9848484848484848,
          "remote": "ms1"
        }
      },
      {
        "title": "Superman: Red Son",
        "genres": [],
        "_federation": {
          "indexUid": "movies",
          "queriesPosition": 1,
          "weightedRankingScore": 0.9848484848484849,
          "remote": "ms0"
        }
      },
      {
        "title": "Superman, Spider-Man or Batman",
        "genres": [
          "Drama"
        ],
        "_federation": {
          "indexUid": "movies",
          "queriesPosition": 1,
          "weightedRankingScore": 0.9848484848484849,
          "remote": "ms0"
        }
      }
    ],
    "processingTimeMs": 15,
    "limit": 5,
    "offset": 0,
    "estimatedTotalHits": 11,
    "facetDistribution": {
      "genres": {
        "Action": 1,
        "Adventure": 1,
        "Comedy": 3,
        "Crime": 2,
        "Documentary": 1,
        "Drama": 1,
        "Family": 1,
        "Fantasy": 1,
        "Horror": 2,
        "Romance": 1,
        "Science Fiction": 1,
        "Thriller": 1,
        "Western": 1
      }
    },
    "facetStats": {},
    "requestUid": "019d05c7-ea65-77a1-8274-22a8ba9e26db",
    "remoteErrors": {}
  }
  

  Note the following to apply the distinct attribute at the federation level:

  1. Applying distinct at the query level at the same time as the federation level is disallowed and will return a HTTP 400 error.
  2. The chosen distinct field will apply to all remotes and indexes, so it must be a filterable attribute for all participating remotes and indexes.
  3. While Meilisearch attempts to compute the most accurate facet distribution, in distributed contexts this cannot be guaranteed as the distinct algorithm is not applied on all of the remote documents.

• Improve performance of federated search by @dureuill in https://github.com/meilisearch/meilisearch/pull/6229

  Improves performance of federated search: about 100ms faster for all requests. Improves reliability of the HTTP server: the server will no longer be blocked when too many federated search requests are being processed.

• Optimize the generation of JSON documents by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6257

  Addresses performance issues that users might encounter when requesting large documents. Additionally, performance is enhanced when users request only a small subset of fields from large documents.

• Use the latest version of mimalloc to improve memory usage by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6201

  Updates mimalloc from v2 to v3, improving memory sharing between threads and significantly reducing memory usage on large workloads. It also overrides the allocator to use mimalloc at linking time, allowing LMDB, Meilisearch, and other C libraries to share their allocations for better overall memory efficiency. @Kerollmops wrote a blog post about the story behind this improvement.

• Add POST /tasks/compact for task queue compaction by @YoEight in https://github.com/meilisearch/meilisearch/pull/6193

  Compacts the task queue database and reclaim space so new tasks can keep being enqueued, without deleting existing tasks. This feature is behind the taskQueueCompactionRoute experimental feature flag.

[!WARNING]
Once task queue compaction completes, all write operations are blocked until the server is restarted.

🔐 Security

• Bump rustls-webpki from 0.103.8 to 0.103.10 in https://github.com/meilisearch/meilisearch/pull/6273
• Bump tar from 0.4.44 to 0.4.45 in https://github.com/meilisearch/meilisearch/pull/6271
• Bump actix-web-lab from 0.24.3 to 0.26.0 in https://github.com/meilisearch/meilisearch/pull/6218
• Bump quinn-proto from 0.11.13 to 0.11.14 in https://github.com/meilisearch/meilisearch/pull/6217

🪲 Bug fixes

• Check char boundaries when hiding secrets by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6262
• Network fixes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6255
• Override "default" source base url with the config one to support AWS bedrock OpenAI API by @dureuill in https://github.com/meilisearch/meilisearch/pull/5905
• Fix panic on virtiofs filesystem by @itsamine27 in https://github.com/meilisearch/meilisearch/pull/6261

🔩 Miscellaneous

• Add X-Accel-Buffering: no for /logs/stream route by @YoEight in https://github.com/meilisearch/meilisearch/pull/6268
• Add link to documentation on --experimental-allowed-ip-networks parameter by @paulden in https://github.com/meilisearch/meilisearch/pull/6263
• Update sysinfo version to 0.38.4 by @GuillaumeGomez in https://github.com/meilisearch/meilisearch/pull/6211
• Add a no-agent AGENTS.md file to prevent excessive AI-generated PRs by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6256

New Contributors

• @GuillaumeGomez made their first contribution in https://github.com/meilisearch/meilisearch/pull/6211
• @itsamine27 made their first contribution in https://github.com/meilisearch/meilisearch/pull/6261