Releases: b2ihealthcare/snow-owl
Snow Owl v8.5.1
Bugs/Improvements
- [index] properly catch and report bulk indexing failures during index commit (e93433c)
- [api] support field selection in SNOMED CT Concept expansions (661c774)
- [api] add update and delete resource REST API endpoints (b79a13c)
- [api] reduce execution time of bulk suggest requests by performing a single authorization check at the beginning (f507964)
- [api] support other resource URIs not just Code Systems in suggest API (64c5559)
- [docs] update rapidoc to latest 9.3.3 version (2dce6a6)
- [docs] minor changes to the Suggest API OpenAPI definition (f550166)
- [log] include auth/authz related execution time in request response metrics (1654004)
Snow Owl v8.5.0
Core
- Introducing the new Code System Concept Suggest API (#1049, 5aad00c)
- Based on lexical and/or semantic information concepts can be retrieved for a given set of like and unlike texts or queries
- Supports pluggable suggester implementations, currently available open-source suggesters are:
term
suggester - based on the most frequent tokens in the given like/unlike corpus the implementation suggests concepts based on their terms using default Elasticsearch full-text queries (synonym support, ignore stopwords, fuzziness, min occurrence count configuration settings are available)mlt
suggester - based on the given most frequent tokens in the like/unlike corpus the implementation suggests concepts based on their terms using Elasticsearch More Like This query (MLT specific configuration values are available, works best for large number of like texts)
- Add
batchSize
andbatchSizeTimeout
configuration parameters toPOST /suggest/bulk
API (05ae0ec)
- Support filtering resources by their settings key-value pairs (#1048, #1050)
- Support Elasticsearch 8 clusters and features via a dedicated high-level client (#1047)
Bugs/Improvements
- [docker] improve docker example (#1044)
- [index] add support for more-like-this Elasticsearch queries (#1049)
- [index] add support for knn Elasticsearch queries (#1049)
- [index] fix decimal value conversion issue (#1039)
- [index] allow configuration of
number_of_shards
andnumber_of_replicas
in custom index specific configurations - [index] ensure that local settings get updated even if there are no remote ES settings to update (5a8d165)
- [index] eliminate deprecation warning messages due to deprecated
ignore_throttled
query parameter in newer Elasticsearch versions (#1053) - [index] increase default
bulkActionSizeInMb
setting value to50
(da69def) - [index] maintain order when merging term filters into a single filter array to ensure that the query is properly cached when requested (df0c594)
- [core] load only the default field selection in
updatedAtCommit()
expansion (baaf59a) - [fhir] fix FHIR date parsing and formatting issue when timezone offset is specified (8e946e8)
- [api] support
active
andversion
parameters incontent()
expand in Resources API (0068a66) - [core] allow querying draft resources with the special
LATEST
path using their current working branch (akaHEAD
) (fa5898d) - [core] add support for ECL query definition inside ResourceURI instances with the
?ecl=
query part (1ded832, 79d112a)
Dependencies/Packaging
- Upgrade Jackson libraries to 2.12.7
- Upgrade Apache HTTP Core to 4.4.12
- Upgrade Apache HTTP Client to 4.5.10
Snow Owl v8.4.0
Core
- Introduce configurable authorization service (#1020, #1026)
- External authorization systems can now be configured to provide list of accessible resource IDs and/or check whether a certain permission is present for a user or not
- The default implementation performs the same evaluation as before based on either the incoming JWT's permissions claim or using information provided by the underlying identity provider
- Introduce a new
GET /toolings
endpoint (#1023)- Lists all supported tooling (aka schema) values from the current deployment
- Support querying resources by their
url
value (#1027) - Support
settings
metadata on Bundle resource types (#1034)
FHIR
- Accept wider range of FHIR inputs based on official R4 and third-party CodeSystem sources (#1017)
- Support
SUPPLEMENT
content mode - Support
code
extension type - Make
div
validator more lenient and accept opening tag without attributes - Support all FHIR-approved date formats in
dateTime
properties - Support FHIR
contact
property (1247e96)
- Support
Bugs/Improvements
- [index] fix incorrectly reduced "must" and "filter" clauses causing invalid search results in certain queries (#1015)
- [index] restrict the number of prefix values and prefix clauses that can be present in a request (eb6df3b)
- [transport] enable eventbus call stack tracing (#1021)
- [core] fix serialization of User object (#1022)
- [core] prevent creating resources with the reserved
-1
identifier (#1036) - [api] support X-Author header in Versioning API (#1018)
- [api] support X-Author header in RF2 Import API (#1031)
- [api] support X-Author header in FHIR CodeSystem API (#1031)
- [api] support filtering resources by
owner
field in Resources API (#1035) - [api] respond with HTTP 401 properly when trying to authenticate with a Snow Owl without any identity provider configured (7f16e9f)
- [fhir] fix potential NPE during lookup property validation (#1019)
- [cli] allow commands to be executed when no JWT signing is configured (b0e81e8)
- [classification] add missing resourcePath parameter to classification jobs (#1025)
- [snomed] fix incorrect RF2 export component type when exporting refsets (#1028)
Dependencies/Packaging
Snow Owl v8.3.0
Core
- Secure connections are now by default enabled when using TCP connections to connect to the terminology server (#993, #1012)
- New configuration values are available to configure SSL certificate (or let the server use a self-signed/generated certificate by default)
- Support Elasticsearch 8 via API compatibility mode (#1008)
- Support running Elasticsearch in testcontainers
- Support externally customizable index type mappings via
repository.index.<indexName>.mappings
configuration key (#1010) - Move ECL labeler API to core (#1009)
SNOMED CT
- Add
batchSize
configuration parameter to SNOMED CT RF2 import API (4109687)
Bugs/Improvements
- [index] backoff a bit when sending too much bulk requests to the connected Elasticsearch cluster on rejection errors, eg. high queue size (be85e38)
- [core] make
toolingId/branch
reflective access support a timestamp suffix (6df066f) - [core] check
bundleId
before executing resource type specific logic (like automatic branch generation) to prevent creation of the corresponding branch as a side effect of an erroneous code system create request (c191ba3) - [core] fetch
extensionOf
branch state from the code system's repository, not the one available from the context. The latter only stores resource metadata and no branches outside MAIN exist in it. (0d581a3) - [core] add a generic implementation of ECL to use in other Code Systems and toolings when required (c5daa92, 76e353d, e6dbf9e)
- [snomed] do not kick off a concept map search in the SNOMED CT tooling if source tooling ID filtering is enabled, and
snomed
is not within the allowed list of tooling IDs. (b30d38d) - [snomed] honor
index.maxTermsCount
when sending large queries to Elasticsearch during RF2 import (4a46daa) - [snomed] reduce load of existing data from external Elasticsearch clusters when importing RF2 releases (cca6f87)
- [snomed] various performance optimizations when sending SNOMED CT commit requests sent to Elasticsearch (f52991a, 62101d6, 26d4a45)
- [suggest] avoid NPE when suggestion base concept does not have alternative terms (0364e6a)
Dependencies
Snow Owl v8.2.1
Snow Owl v8.2.0
Core
- Add new Base62 encoded UUID generator (93c1711)
- Replace all uses of Base64 encoded UUID with Base62 encoded UUIDs (f345312)
SNOMED CT
- Support ECL 2.0 expressions to be parsed and evaluated, see official spec here (#997)
- Support multi-valued string evaluation in concrete values
- Support
!=
operator inactive
andmoduleId
property filters - Support
active
/moduleId
/effectiveTime
property filters with member domain - Support member field property filters
- Support history supplements
- Selecting refsetFieldNames in
memberOf
expression is unsupported (anything that is not thereferencedComponentId
at the moment) - Support ignoring ECL specific validation warnings/errors when parsing expressions
Bugs/Improvements
- [core] add guards to prevent wrapping single collection values again in another collection in HashMapOptions (3acf6ec)
- [ecl] fix case when
AND
expression constraint has a single ID only (50e943d) - [api] add support for explicit
referencedComponentId
filtering in SNOMED CT Member API prop filter (e10fff0) - [api] properly set default values in resource create API even if properties exist with
null
values in the incoming JSON body (1dd0a54) - [scripts] wrap groovy 3.0.9 jars into a bundle to avoid dependency issues in third party plug-ins (a3a5bd6)
Dependencies
- Bump snomed-ecl to 2.0.3
- Add uuid-creator 4.6.1
Snow Owl v8.1.1
Snow Owl v8.1.0
Breaking changes
The following changes in Snow Owl 8.1 might affect your applications and prevent them from operating normally. Before upgrading to 8.1, review these changes and take the described steps to mitigate the impact.
Java 17
Snow Owl 8.1 brings complete Java 17 support, both compile and runtime. This mostly affects users who manually install Snow Owl or use a development environment to develop plug-ins for it. Starting from 8.1.0, all Snow Owl release include the latest patch version of Eclipse Temurin JDK, which increases the package size slightly, but installation/deployment is simpler. See PR #963 for details.
Core
- Support retrieval of historical state for resources (#990)
- Support
resourcePathSegments
property andresourcePathLabels
expansion on resources. This allows clients to get information about the placement of the resource in the bundle tree. (#961) - Support expansion of
commits()
in Resource API (#966) - Improved resource history information
- Support proper API doc navigation via query parameters (#980)
- Support pluggable resource expansion for third party plug-ins and features (#967)
SNOMED CT
Security
- OAuth 2.0 authorization token features (#937):
- Support HS256, RS256 and RS512 algorithms (verification all, signing only HS256 and the existing HS512).
- Support external JWKS URL for token verification
- Support configuration of
email
andpermissions
JWT token claims to allow OIDC conformant JWT token verification. (claims have their own namespace, likehttps://example.com/email
,https://example.com/permissions
). Legacy token handling is still the default, where the email property is stored in thesub
claim, while the permission list is stored in thepermissions
claim. - Support PKCS#8 signing and verification keys
- Disable JWT token signing/verification if
identity.jws
is not configured.
Bugs/Improvements
- [index] optimize bool index queries as much as possible to prevent HTTP 400 errors because of deep nested queries (configurable in Elasticsearch via
indices.query.bool.max_nested_depth
) (#976) - [index] filter out hidden indices to prevent issues when checking cluster health status (#958)
- [index] fix incorrectly detected read-only state when
.index.blocks.read_only_allow_delete
setting is set (3d829aa) - [index] add support for immediate (partial) updates via scripts (#989)
- [monitor] add cluster health diagnosis message to cluster health errors (f1bc20c)
- [core] connect id filter in
GET /versions
endpoint (43c7992) - [core] carry over parentage data to generic Concept model (#950)
- [core] make sure we track scores when filtering resources by title (299258a)
- [core] filter commit messages by the current user's permission list (#984)
- [core] include resource title in default delete commit message
- [core] add generic taxonomy change processor that calculates a transitive closure graph on each commit based on the changes (#981)
- [auth] prevent access to resources when permissions claim is empty for the current user or in the given JWT accessToken (#971)
- [fhir] allow filtering fhir resources by
url
in theid
filter (01a6885) - [fhir] make sure fields configured via _summary are returned even if user explicitly specifies additional fields via
_elements
(e75623e) - [fhir] allow returning map targets for any target code systems (e14af6f)
- [fhir] support
datetime
type in concept properties (fd56483) - [fhir] validate system URL against version given in subsumption requests (#982)
- [fhir] add abstractions that allow third party developers to implement FHIR Create/Update/Delete support (#968)
- [snomed] fixed an issue when getting namespace from concept with at least two different FSNs (translations) (e215814)
- [snomed] defer loading of expanded additional data to dedupe requests going to Elasticsearch and reduce load on both the ES cluster and the termserver (#948)
- [snomed] add missing expand and field selector to relationship GET by ID endpoint (#949)
- [security] mitigate CVE-2021-44228 vulnerability (a8127da)
- [security] mitigate CVE-2021-42550 vulnerability (69b710d)
Packaging
- Update bundled JDK to 17.0.2
Dependencies
- Bump Elasticsearch to 7.17.1
- Bump Groovy to 3.0.9
- Bump Xtend and Xtext to 2.25.0
- Bump MWE to 2.12.1
- Bump rapidoc to 9.2.0
- Bump typetools to 0.6.3
- Bump mockito to 3.11.2
- Bump jacoco to 0.8.7
- Bump rest-assured to 4.3.0
- Bump Tycho to 2.6.0
Snow Owl v8.0.0
Snow Owl's biggest release yet comes with fully customizable resource model and complete resource management functionality; the fastest RF2 importer and exporter in the world; externally configurable per resource access management; latest Expression Constraint Language 1.6 features and many many more.
Breaking changes
The following changes in Snow Owl 8.0 might affect your applications and prevent them from operating normally. Before upgrading to 8.0, review these changes and take the described steps to mitigate the impact.
RF2 index format changes
To support sorting, filtering and field selection on the official SNOMED CT RF2 property names, Snow Owl 8 comes with an updated index representation where each and every SNOMED CT component index has correct, officially supported RF2 column names. Migrating from Snow Owl 7.x to 8.0 requires a full re-import of all terminology resource content.
API routing changes
Snow Owl 8 changes the old /admin
and /snomed-ct/v3
Core and SNOMED CT REST API base path segments to /core
and /snomedct
, respectively. Make sure you update your REST API clients when you start your migration process to the new version. See PR #893 for more details.
Update endpoint changes
Some of the old SNOMED CT Component endpoints were using a POST /:componentType/updates
style instead of PUT /componentType
. Snow Owl 8 migrated all API endpoints using the former pattern to the new one. Make sure you update your REST API clients to the new pattern, if you have used any of the SNOMED CT component update API endpoints. For more details, please see PR #866.
Configuration changes
- Snow Owl does not support node-level configuration of certain SNOMED CT specific settings. These all have been moved to Code System resource specific configuration keys and can be configured for each SNOMED CT specific Code System separately (eg. for each Extension). The affected configuration settings are:
Removed custom MRCM model
To support variety of use cases and easier adoption of SNOMED CT MRCM constraints, Snow Owl from version 8 fully supports not just MRCM member access and authoring, but evaluation of official and custom (maintained by NRCs and Extensions) MRCM constraints in the fastest way possible. The old custom MRCM model and all related API functionality has been removed in favor of the official MRCM tooling. See PRs #917 and #930 for more details.
Removed scroll based paging
Scroll API in Elasticsearch has been marked as deprecated until complete removal in the upcoming Elasticsearch 8.0 release. Thus, Snow Owl is no longer offering low-level Java and index APIs to access the terminology content via scroll contexts. For more information see PR #903.
Core
- Introducing Resource Management (#821, #886)
- Any type of resources (when the appropriate model and API is available, plug-ins can provide their own specific models if they so desire)
- Supported resources are
Code System
,Bundle
(and other resources such asValue Set
s andConcept Map
s are available in pro versions) - Full REST API support to access (get, search), author (create, update, delete) and maintain (versioning, configuration, history) resources
- New Core API features:
- New concept suggest API (#878, #882, #884, #921)
- Field selection support (#858)
- Restricting the number of returned fields of a representation to a set of properties is now possible on read endpoints (e.g
GET /snomedct/SNOMEDCT-UK-CL/concepts?field=id
) - By default the system will always return all properties with a non-null value, but this can be altered using the
field
query parameter.
- Restricting the number of returned fields of a representation to a set of properties is now possible on read endpoints (e.g
- Support
includeNull
query parameter to allow API consumers to select whether they would like to get all properties (with any value, includingnull
) or just the non-null ones (default setting) (#857) - Support
pretty
query parameter to pretty print the JSON output to make it human-readable (#857) - Support multi-value expand parameters on all search/get endpoints (#855, fixes #659)
- Support filtering versions by
createdAt
andversion
(#874, cdae9ac) - Support Forwarded headers on incoming HTTP requests when needed (cc7c544)
- Support deletion of Code Systems via
DELETE /codesystems/:id
(d416c1e) - Use
yyyy-MM-dd
effective time format when creating new versions (27e34ab) - Increase branch path segment max length to
100
characters from the previous50
characters (2fe892e)
- New Index features:
- Reduce the default
number_of_shards
to one as Elasticsearch offers the same default,number_of_replicas
still defaults to 0 to support initial loading scenarios better (fdd10ce, 6b1d462) - Allow fully customizable indices settings in
snowowl.yml
configuration file viarepository.index.[indexName]
configuration objects (#929) - Customizable word equivalence (aka synonym) filtering (#863)
- An
SO_HOME/configuration/analysis/synonym.txt
file can be configured to allow Snow Owl to use the defined set of synonym rules when performing full-text searches. In case of remote Elasticsearch cluster, it is required to deploy thesynonyms.txt
on all cluster nodes
- An
- Support index-level search request caching (#931)
- Any kind of search can be cached and retrieved faster than ever before. Plug-in maintainers and new tool developers can configure their preference of how and when they'd like to cache the results of a search request. By default Snow Owl will cache all search requests that are executed on versioned state.
- Support numeric script sorting in index searches (8e51c6b)
- Reduce the default
- Deprecated branch review API endpoints and services have been removed (b79c8e1)
SNOMED CT
- New SNOMED CT API features:
- Support dedicated
isActiveMemberOf
filter which return concepts that has an active member in any of the given list of refsetIds (#879) - Support searching SNOMED CT components by
namespaceConceptId
not justnamespace
(#881, #920) - Support exporting Snapshot RF2 content at specific times via
resourceId@timestamp
format,Full
andDelta
are not supported (#887) - Support exporting RF2 content between specific branch path range, like comparing branches (#907)
- Support nested expansion of
inactivationIndicator()
properties, likeassociationTarget()
,referenceSet()
(#914) - Support sorting when expanding
descriptions()
,relationships()
,inboundRelationships()
on Concept API (7e2516a) - Expose
memberOf
andactiveMemberOf
components document fields on component domain representations (5928462) - Support component type (
concept
,description
,relationships
,member
) based filtering on RF2 export endpoint (c8662e4) - Support multi-value query parameters on a lot of filter query parameters (5e5fc2a)
- Support
module
,active
andeffectiveTime
filters onGET /refsets
endpoint (5e5fc2a)
- Support dedicated
- RF2 import improvements (#918)
- Support smart auto-generation of delta content when importing Full releases (the existing component-based effective time based filtering is still there, but this makes the import much faster and easier to understand)
- Support
importUntil
effective time property to import only a portion from an RF2 release - Disallow importing RF2 releases with
createVersions
enabled when the target branch is a CodeSystem's main development branch and it has unpublished changes - Automatically configure known language dialect aliases after successful RF2 import (#876)
- Consolidate RF2 property names on index documents to allow sorting, filtering on them using official RF2 column names (#856)
- Update ECL to 1.6 features (#837, #915)
- The Query Language Draft implementation has been removed and extra unofficial features of it have been merged entirely into the ECL syntax. The SNOMED CT Concept search REST API endpoint's
query
parameter has also been removed. If you had any query language specific expressions, you might need to migrate them to the new ECL 1.6 syntax which includes all previously supported QL features.
- The Query Language Draft implementation has been removed and extra unofficial features of it have been merged entirely into the ECL syntax. The SNOMED CT Concept search REST API endpoint's
- Deprecated SNOMED CT
/imports
and/exports
endpoints have been removed (82938f6)
Security
- Improved authorization options to allow/restrict access to server resources. Access tokens describe the necessary scopes to give access to certain resources, either by:
- directly denoting the resource ID in the scope
- denoting the containing bundle ID in the scope (supports direct and indirect parent bundles)
- or by allowing access to all resources (*)
rootDn
androotDnPassword
support from ldap security realm configuration have been removed (6483870)
FHIR
- Support
CodeSystem
,ValueSet
andConceptMap
filtering by_id
,name
,title
,system
,url
,version
and other useful FHIR properties that are supported in the underlying resource representation (#909) - Support for CapabilityStatement and OperationDefinition resources (#899)
- Read-only Bundle API support (#867)
- Experimental Batch API support (#867)
- Revised and improved all existing FHIR API endpoints to resolve all outstanding issues, such as (#416, #426, #435, #494, #561)
- Support
after
cursor based paging when expanding Value Sets (e9431f9)