Skip to content

Discard Microsoft.Data.SqlCLient.SqlMetaData.xml in favor of storing metadata internally - #4362

Open
tetolv wants to merge 32 commits into
dotnet:mainfrom
tetolv:getschema-rewrite
Open

tetolv wants to merge 32 commits into
dotnet:mainfrom
tetolv:getschema-rewrite

Conversation

@tetolv

@tetolv tetolv commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Description

Continuation of @paulmedynski effort on #3372 to pull inside one by one all metadata collections from Microsoft.Data.SqlCLient.SqlMetaData.xml. Here it led me to complete rewrite of the SqlMetaDataFactory. Main reasons were:

  • Replacing DataTable as cached representation of metadata, with dedicated classes, because they require less checks for data consistency, they should consume less memory and we don't require DataTable querying capabilities anyway
  • More streamlined design. Different metadata type collection are encapsulated in their own classes and it's possible to more easily extend it with new kinds of metadata, although I don't think is it's necessary.
  • All the actual static metadata (in opposite to metadata read from server system views) is kept in a single structure s_metaDataCollection , which in most cases should be the only place, where new metadata additions and tweaks should be performed, without need to dig deep into the code, which handles this data structure. There I have diverted from the code style guidelines to use longer lines, because that structure by nature has rather tabular representation and it would never fit into 100 characters, but I think that maintaining it's tabular representation is more important to code readability.

Fixes

  • sys.table_types, used in query for StructuredTypeMembers metadata collection was supported since SQL2008(v10.0), not since SQL2005(v9.0). I have updated minimumVersion accordingly.

Localization

MDF_DataTableDoesNotExist need to be rephrased and translated. Original text was The collection '{0}' is missing from the metadata XML., but obviously it is not valid anymore.

Testing

I have compared metadata returned from rewritten version of SqlMetaDataFactory with metadata returned from MDS v7.0.1. I have attached comparison results in SchemaComparison.zip.
SchemaComparison.zip It has the following structure:

---
config:
    treeView:
        rowIndent: 10
        lineThickness: 1
    themeVariables:
        treeView:
            labelFontSize: '15px'
            labelColor: '#808080'
            lineColor: '#606060'
---
treeView-beta
    "09.00.5000  (MSSQL 2005)"
        "legacy"
             "multiple .csv files for each collection."
        "new"
             "multiple .csv files for each collection"
    "12.00.9114 (Azure database)"
        "legacy"
             "multiple .csv files for each collection"
        "new"
             "multiple .csv files for each collection"
    "17.00.4015 (MSSQL 2025)"
        "legacy"
             "multiple .csv files for each collection"
        "new"
             "multiple .csv files for each collection"
Loading

Here's a list of all notable differences:

  • In ReservedWords collection in all versions (09.00.5000, 12.00.9114, 17.00.4015) all elements are rearranged, but no new items appears, or were removed. Rearrangement is the same as in #PR4328, because I took a list of reserved words from there.
  • In Restrictions collection in all versions, there exists a column RestrictionDefault. But in https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/common-schema-collections it is marked as Ignored. Previously for all the metadata collections, which were based on query to server system views, this column contained name of the column in system view, which was used by the restriction, but it was not always the case. For now I have decided to do not return anything in RestrictionDefault column, but if it's deemed important, then I can return there the same values as before.
  • In Restrictions collection, filtering by server engine version wasn't working correctly, so on v09.00.5000 restrictions for collections AllColumns, ColumnSetColumns, StructuredTypeMembers were returned, although collections itself are not supported on that engine version.
  • In MetaDataCollections there are two new collections TVPs and UDTs. Those are used by DataTypes collection, where in addition to static data types, there are also returned table-valued parameter types, and user-defined types, that are read from the server system views. Previously there were separate methods for querying each of them and appending to the DataTypes collection, but now they are handled by separate SqlCommandCollection instances added to the s_metaDataCollection. As result are also returned in MetaDataCollections. Those two could either remain as undocumented collections, or I can somehow mark them as internal, so they wouldn't be returned in MetaDataCollections.

No automatic tests were added, because affected functionality is completely private.

Will be happy to hear any feedback.

…d of Microsoft.Data.SqlClient.SqlMetaData.xml
@tetolv
tetolv marked this pull request as ready for review June 15, 2026 12:49
@tetolv
tetolv requested a review from a team as a code owner June 15, 2026 12:49

@edwardneal edwardneal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. I've made a first-pass review for discussion.

"DUMMY"
]),
new SqlCommandCollection("Users", 1, 1, null, null, "select uid, name as user_name, createdate, updatedate from sysusers where (name = @Name or (@Name is null))",
[new Restriction(1, "User_Name", "@Name")]),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to specify the sequence number here? I can't see a reason why we'd ever want to have duplicates.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, I understand your question. The reason I left explicit restriction numbers there, is that according to documentation https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/common-schema-collections#restrictions , column RestrictionNumber is required and for compatibility with the original ordering of the restrictions, I made those numbers explicit. Of course they could be removed and resulting table would rely on the implicit restrictions order in array for each sql-based collection.

private readonly string _populationString;
public readonly Restriction[] RestrictionParams;

public SqlCommandCollection(string collectionName, int numberOfRestrictions, int numberOfIdentifierParts,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are no circumstances where numberOfRestrictions is not restrictions.Length then we don't need to specify it explicitly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just that there are two brotherly columns NumberOfRestrictions and NumberOfIdentifierParts, which are explicitly required by documentation https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/common-schema-collections#metadatacollections and they have a direct dependency: NumberOfIdentifierParts could not be greater than NumberOfRestrictions. If we would separate them, by replacing explicit definition of NumberOfRestrictions with a length of restriction array, then this dependency would not be so obvious for anyone adding new query-based collections.
Also I wanted for metadata definition in code, to resemble as close as possible, metadata returned to the caller.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that relationship truly guaranteed? It's not laid out in the documentation linked, and if we look at the IndexColumns view, we see five restrictions: Catalog, Owner, Table, ConstraintName, Column. An index column doesn't even have an identifier, so there's no relationship to maintain in that situation. It's the same story with Procedures - restriction 4 is Type, and this doesn't form part of the identifier.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems, that I completely misunderstood purpose of NumberOfIdentifierParts... Yes, it's not related to NumberOfRestrictions in any way, so explicit NumberOfRestrictions could be substituted with deriving it's value from the actual number of restrictions in SqlCommandCollection.RestrictionParams collection.
Done.

// 7. Variable-length string and binary types: varchar, nvarchar, varbinary.
// 8. Miscellaneous fixed-length types: uniqueidentifier, sql_variant, timestamp

AddFixedNumericType(SqlDbType.TinyInt, isBestMatch: true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've lost a lot of metadata by replacing these with a single list; the goal of doing this was to make it difficult for DataTypes to drift from SqlClient's internal MetaType definitions.

Can we replace the list in SqlMetaDataFactory.cs with multiple constructors, and derive data from MetaType definitions in the same way?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have thought about unification with MetaData definition, and even initially had a second constructor for SqlMetaDataFactory.TypeMetaData which would use them, but very few fields would match directly one-to-one between the two definitions for each type, so I decided to go on with a flat table, where all type metadata are defined explicitly. I bear in mind, that in #3372 there was an additional proposal to make general unification of all datatype info in one place, but right now I thought it to be a task for later.
Right now I have settled on the intermediate solution, I initialize all types in the private static readonly TypeMetaData[] s_types with a constructor, that takes SqlDbType and derive from it all the type values, that have a direct one-to-one relation to database schema datatype values,

        public TypeMetaData(SqlDbType dbType, ...)
        {
            // Shared type properties
            MetaType metaType = MetaType.GetMetaTypeFromSqlDbType(dbType, isMultiValued: false);
            TypeName = alias ?? metaType.TypeName;
            ProviderDbType = (int)metaType.SqlDbType;
            DataType = metaType.ClassType.FullName!;

            // Schema specific type properties
            ...

, and the rest are defined explicitly in s_types as before.
The reason I haven't taken many-constructor approach is, because there were constructors, used by several datatypes, but inside a constructor there were conditional logic based on particular datatype being constructed. Not only this logic was hard to follow to create a complete mental picture of what properties of constructed datatype would be, but also adding a new data type would anyway require either adding a new dedicated constructor, or additionally enhance inner logic of one of the existing constructors.

When datatype metadata unification will eventually be made, then I expect that datatype class will have all the fields required by all datatype representations that we have, and optionally perhaps a set methods: ToMetaType, ToSchemaRow, etc..., to produce particular type representations where needed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have additionally updated my version, so now almost everything about data types is derived from Microsoft.Data.SqlClient.MetaType, except for MaximumScale property for date, datetime2, datetimeoffset.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a bit of thought, I'm still inclined to say that we should retain the style of logic in AddFixedNumericType, rather than hardcoding IsNullable, etc. as hardcoded parameters. The MetaType class is the core of SqlClient's type logic; it shouldn't need to concern itself with the implementation details of the other parts of the library which sit downstream and consume its type information. This pushes me away from having ToSchemaRow / ToTypeMetaData / etc. methods, and back towards forcing TypeMetaData to derive its information wholly from MetaType.

If constructors would be unclear, perhaps we could have something similar to:

TypeMetaData smallInt = TypeMetaData.FixedNumericType(SqlDbType.SmallInt, isBestMatch: true);
TypeMetaData time = TypeMetaData.VariablePrecisionDateTimeType(SqlDbType.Time, columnSize: 5,
    isBestMatch: false, minimumVersion: "10.00.000.0");

This would mean that we'd have clearly-defined methods with their own logic for generating the correct values.

@tetolv tetolv Jul 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have substituted tabular representation of the date type list with the separate methods for each data type group, similar as it was before. But anyway the amount of type data, that is taken from Microsoft.Data.SqlClient.MetaData remain the same. IsNullable and most of the other attributes are not taken from there, only TypeName, ProviderDbType, DataType, ColumnSize, CreateFormat are.

While refactoring that, I have noticed several other areas, of which I am unsure:

  • For several data types: timestamp, binary, image, text, ntext, xml, varchar, char, nchar, nvarchar, varbinary, in column IsLiteralSupported original version, which used XML file, returned empty string, but with your PR, it started to return True. On one hand it's consistent with values in columns LiteralPrefix and LiteralSuffix. If they return anything, then literals must be supported on this data type. On the other, doesn't it break backward compatibility?
  • Similar question could be raised about all date/time data types, which all have LiteralPrefix and LiteralSuffix defined, but this time IsLiteralSupported=(empty)
  • Both date and time data types have LiteralPrefix={ ts ', but according to https://learn.microsoft.com/en-us/sql/t-sql/data-types/datetime-transact-sql?view=sql-server-ver17#odbc-format , or https://learn.microsoft.com/en-us/sql/odbc/reference/appendixes/date-time-and-timestamp-escape-sequences?view=sql-server-ver17 they should have d and t prefixes respectively. Is ts prefix for them is intentional, or is it mistake? Should literal prefix be updated for those data types, or left as-is? I have checked it via
    DECLARE @query nvarchar(MAX) = 'select { ts ''1998-05-02 01:23:56.123'' },{ d ''1990-10-02'' },{ t ''13:33:41'' }';
    EXEC sys.sp_describe_first_result_set @query, NULL, 0;
    
    and for all three expressions, it returns data type datetime, so in fact prefixes d and t doesn't have any effect on setting proper literal type.

Here for a reference is the result of calling GetSchema (I have also retained both UDTs and TVPs in my sample DB, just in case):

TypeName ProviderDbType ColumnSize CreateFormat CreateParameters DataType IsAutoIncrementable IsBestMatch IsCaseSensitive IsFixedLength IsFixedPrecisionScale IsLong IsNullable IsSearchable IsSearchableWithLike IsUnsigned MaximumScale MinimumScale IsConcurrencyType IsLiteralSupported LiteralPrefix LiteralSuffix
smallint 16 5 smallint System.Int16 True True False True True False True True False False False
int 8 10 int System.Int32 True True False True True False True True False False False
real 13 7 real System.Single False True False True False False True True False False False
float 6 53 float({0}) number of bits used to store the mantissa System.Double False True False True False False True True False False False
money 9 19 money System.Decimal False False False True True False True True False False False
smallmoney 17 10 smallmoney System.Decimal False False False True True False True True False False False
bit 2 1 bit System.Boolean False False False True False False True True False False
tinyint 20 3 tinyint System.Byte True True False True True False True True False True False
bigint 0 19 bigint System.Int64 True True False True True False True True False False False
timestamp 19 8 timestamp System.Byte[] False False False True False False False True False True True 0x
binary 1 8000 binary({0}) length System.Byte[] False True False True False False True True False False True 0x
image 7 2147483647 image System.Byte[] False True False False False True True False False False True 0x
text 18 2147483647 text System.String False True False False False True True False True False True ' '
ntext 11 1073741823 ntext System.String False True False False False True True False True False True N' '
decimal 5 38 decimal({0},{1}) precision,scale System.Decimal True True False True False False True True False False 38 0 False
numeric 5 38 numeric({0},{1}) precision,scale System.Decimal True True False True False False True True False False 38 0 False
datetime 4 23 datetime System.DateTime False True False True False False True True True False {ts ' '}
smalldatetime 15 16 smalldatetime System.DateTime False True False True False False True True True False {ts ' '}
sql_variant 23 sql_variant System.Object False True False False False False True True False False False
xml 25 2147483647 xml System.String False False False False False True True False False False False
varchar 22 2147483647 varchar({0}) max length System.String False True False False False False True True True False True ' '
char 3 2147483647 char({0}) length System.String False True False True False False True True True False True ' '
nchar 10 1073741823 nchar({0}) length System.String False True False True False False True True True False True N' '
nvarchar 12 1073741823 nvarchar({0}) max length System.String False True False False False False True True True False True N' '
varbinary 21 1073741823 varbinary({0}) max length System.Byte[] False True False False False False True True False False True 0x
uniqueidentifier 14 16 uniqueidentifier System.Guid False True False True False False True True False False ' '
date 31 3 date System.DateTime False False False True True False True True True False {ts ' '}
time 32 5 time({0}) scale System.TimeSpan False False False False False False True True True 7 0 False {ts ' '}
datetime2 33 8 datetime2({0}) scale System.DateTime False True False False False False True True True 7 0 False {ts ' '}
datetimeoffset 34 10 datetimeoffset({0}) scale System.DateTimeOffset False True False False False False True True True 7 0 False {ts ' '}
json 35 2147483647 json System.String False False False False False True True False False False True ' '
"Microsoft.SqlServer.Types.SqlHierarchyId,Microsoft.SqlServer.Types,Version=17.0.0.0,PublicKeyToken=89845dcd8080cc91" 29 892 True True False
"Microsoft.SqlServer.Types.SqlGeometry,Microsoft.SqlServer.Types,Version=17.0.0.0,PublicKeyToken=89845dcd8080cc91" 29 -1 True True False
"Microsoft.SqlServer.Types.SqlGeography,Microsoft.SqlServer.Types,Version=17.0.0.0,PublicKeyToken=89845dcd8080cc91" 29 -1 True True False
PipelineStatsType 30 -1 False False False
PipelineType 30 -1 False False False
StageType 30 -1 False False False
AdapterType 30 -1 False False False
InstanceType 30 -1 False False False

@tetolv

tetolv commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor Author

I have several more questions:

  • should vector datatype be added now, or later?
  • I'm usually bad at naming things, nor I understand terminology used in MDS perfectly, so if any of my classes/methods/properties sounds unclear, or confusing, feel free to suggest renames for them.
  • There are bunch of new small classes/interfaces stored internally under SqlMetaDataFactory, they all seems quite messy. How would it better to reorganize them?
  • One error text MDF_DataTableDoesNotExist (The collection '{0}' is missing from the metadata XML.) should be renamed and translated. How to do that?
  • Which automatic tests might be needed?

@benrr101

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

@paulmedynski paulmedynski moved this from To triage to Backlog in SqlClient Board Jul 16, 2026

@edwardneal edwardneal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tetolv. Sorry for taking a while to pick up. ConnectionCapabilities has now been merged, would you mind updating your branch please?

// 7. Variable-length string and binary types: varchar, nvarchar, varbinary.
// 8. Miscellaneous fixed-length types: uniqueidentifier, sql_variant, timestamp

AddFixedNumericType(SqlDbType.TinyInt, isBestMatch: true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a bit of thought, I'm still inclined to say that we should retain the style of logic in AddFixedNumericType, rather than hardcoding IsNullable, etc. as hardcoded parameters. The MetaType class is the core of SqlClient's type logic; it shouldn't need to concern itself with the implementation details of the other parts of the library which sit downstream and consume its type information. This pushes me away from having ToSchemaRow / ToTypeMetaData / etc. methods, and back towards forcing TypeMetaData to derive its information wholly from MetaType.

If constructors would be unclear, perhaps we could have something similar to:

TypeMetaData smallInt = TypeMetaData.FixedNumericType(SqlDbType.SmallInt, isBestMatch: true);
TypeMetaData time = TypeMetaData.VariablePrecisionDateTimeType(SqlDbType.Time, columnSize: 5,
    isBestMatch: false, minimumVersion: "10.00.000.0");

This would mean that we'd have clearly-defined methods with their own logic for generating the correct values.

private readonly string _populationString;
public readonly Restriction[] RestrictionParams;

public SqlCommandCollection(string collectionName, int numberOfRestrictions, int numberOfIdentifierParts,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that relationship truly guaranteed? It's not laid out in the documentation linked, and if we look at the IndexColumns view, we see five restrictions: Catalog, Owner, Table, ConstraintName, Column. An index column doesn't even have an identifier, so there's no relationship to maintain in that situation. It's the same story with Procedures - restriction 4 is Type, and this doesn't form part of the identifier.

Comment on lines +358 to +360
public bool SupportedByCurrentVersion(string serverVersion) =>
(_minimumVersion == null || string.Compare(serverVersion, _minimumVersion, StringComparison.OrdinalIgnoreCase) >= 0) &&
(_maximumVersion == null || string.Compare(serverVersion, _maximumVersion, StringComparison.OrdinalIgnoreCase) <= 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking on this. I'd been waiting for the ConnectionCapabilities PR to merge before coming back, and I think we're working along the same lines.

ConnectionCapabilities is a purely passive class - it only responds to the capabilities which the server sends us, and so wouldn't know the engine edition. In my view, it should stay that way.

Azure is detected using ConnectionCapabilities.IsAzureSql, but don't worry about adding the new logic for the json / vector types - it's best to keep this PR focused on the removal of the XML resource file. Once this PR's reviewed and merged, a follow-up can handle the new types.

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to inactivity for more than 30 days.

If you would like to keep this pull request open, please provide an update or respond to any comments. Otherwise, it will be closed automatically in 7 days.

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to inactivity for more than 30 days.

If you would like to keep this pull request open, please provide an update or respond to any comments. Otherwise, it will be closed automatically in 7 days.

@github-actions github-actions Bot added the Stale The Issue or PR has become stale and will be automatically closed shortly if no activity occurs. label Oct 24, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

This pull request has been automatically closed due to inactivity. It was labeled as stale, and has not been updated since then.

If you believe this pull request is still relevant and needs attention, please feel free to reopen it or create a new pull request with updated changes.

Thank you for your contribution to the Microsoft.Data.SqlClient project!

@github-actions github-actions Bot closed this Nov 1, 2026
@github-project-automation github-project-automation Bot moved this from Backlog to Done in SqlClient Board Nov 1, 2026
@paulmedynski paulmedynski reopened this Nov 2, 2026
@github-project-automation github-project-automation Bot moved this from Done to Backlog in SqlClient Board Nov 2, 2026
@github-actions github-actions Bot removed the Stale The Issue or PR has become stale and will be automatically closed shortly if no activity occurs. label Nov 3, 2026

@priyankatiwari08 priyankatiwari08 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Scope: 9c952ab; source re-review of metadata, resources, and build changes.

Existing feedback: Regression tests and smaller PRs are still outstanding. The old draft's speculative and pre-existing concerns have been removed.

Findings: Two regressions, detailed inline:

  • P1: Generated string lookups use the wrong resource manifest name.
  • P2: Static schema collections and appended UDT/TVP rows lose their previous Unchanged state.

Coverage: Add sync/async regression checks for resource lookup and schema row states; existing schema tests do not assert these cases.

Validation: Product tests not run locally (source-only review); driver CI is blocked by merge conflicts.

@github-project-automation github-project-automation Bot moved this from Backlog to Waiting for customer in SqlClient Board Nov 3, 2026
@tetolv

tetolv commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

I have updated this branch with the latest main, which required some non-standard changes regarding support for vector data type, including changes in tests.
During review you might want to pay closer attention to that.

@cheenamalhotra cheenamalhotra added the Localization Needed PRs that contain changes that would need localization label Nov 6, 2026

@edwardneal edwardneal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got a few comments - some of them nits, some of them the result of #4881.

The vector methodology is awkward, but I don't think there's much to be done in this PR - we're parked downstream of some odd behaviour in the underlying metadata selection.

@@ -170,748 +140,125 @@ public async Task<DataTable> GetSchemaAsync(DbConnection connection, string coll

public async ValueTask<DataTable> GetSchemaCore(DbConnection connection, string collectionName, string[] restrictions, bool isAsync, CancellationToken cancellationToken)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Largely unrelated nit: GetSchemaCore, GetSchemaAsync and GetSchema no longer need to access instance data. This entire class could become static in theory, which means that each connection pool group could shed its SqlMetaDataFactory instance.

I'm not asking you to do anything with this - just want to raise it for wider discussion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now I'm wondering we we need to have SqlMetaData as a factory at all. As I understand, initial intent was to read metadata XML into that factory once, and then serve from it all GetSchema requests. But now since all the metadata is static, factory doesn't store anything on instance level and could be bypassed.

Comment on lines +113 to +117
new SqlCommandCollection("_TVPs", 2,
[new SupportedQuery("10.00.0000", null, "select name TypeName, 30 ProviderDbType, max_length ColumnSize, null CreateFormat, null CreateParameters, null DataType, null IsAutoincrementable, null IsBestMatch, null IsCaseSensitive, null IsFixedLength, null IsFixedPrecisionScale, null IsLong, is_nullable IsNullable, 0 IsSearchable, null IsSearchableWithLike, null IsUnsigned, null MaximumScale, null MinimumScale, null IsConcurrencyType, 0 IsLiteralSupported, null LiteralPrefix, null LiteralSuffix, null NativeDataType from sys.types where is_table_type = 1")],
[]),
new SqlCommandCollection("_UDTs", 2,
[new SupportedQuery("09.00.0000", null, "select types.assembly_class COLLATE database_default + ', ' + assemblies.name + ', Version=' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionMajor')) + '.' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionMinor')) + '.' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionBuild')) + '.' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionRevision')) + ISNULL(', Culture=' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'CultureInfo')),'') + ISNULL(', PublicKeyToken=' + LOWER(REPLACE(CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'PublicKey'),1),'0x','')),'') TypeName, 29 ProviderDbType, max_length ColumnSize, null CreateFormat, null CreateParameters, null DataType, null IsAutoincrementable, null IsBestMatch, null IsCaseSensitive, is_fixed_length IsFixedLength, null IsFixedPrecisionScale, null IsLong, is_nullable IsNullable, 1 IsSearchable, null IsSearchableWithLike, null IsUnsigned, null MaximumScale, null MinimumScale, null IsConcurrencyType, 0 IsLiteralSupported, null LiteralPrefix, null LiteralSuffix, null NativeDataType from sys.assemblies as assemblies join sys.assembly_types as types on assemblies.assembly_id = types.assembly_id")],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the removal of SQL 2005 support, do we still need these (and the associated accumulator support, and the special-casing of _ in SqlMetaDataFactory.MetaDataCollection.GetMetadata) or can we simplify this further?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can do that, and it will somewhat simplify code, but at the same time it depends on a previous question of whether we should remove Predicate<ConnectionCapabilities>? IsSupported predicates for each collection, because after dropping support for SQL2005 all current collections seems to be supported unconditionally. If team will decide to leave support for that predicate, then we better leave here original implementation, which calls FindMetaDataCollection("_UDTs", context), or FindMetaDataCollection("_TVPs", context);, because it's internally checks that predicate.

How it looks now:

            // 2. Add UDTs from the server if supported
            MetaDataCollectionBase? udtCollection = FindMetaDataCollection("_UDTs", context);
            if (udtCollection != null)
            {
                const string GetEngineEditionSqlCommand = "SELECT SERVERPROPERTY('EngineEdition');";
                using SqlCommand command = ((SqlConnection)context.Connection).CreateCommand();

                command.CommandText = GetEngineEditionSqlCommand;
                int engineEdition = (int)(context.IsAsync ? await command.ExecuteScalarAsync(context.CancellationToken).ConfigureAwait(false) : command.ExecuteScalar());
                // Azure SQL Edge (9) throws an exception when querying sys.assemblies
                // Azure Synapse Analytics (6) and Azure Synapse serverless SQL pool (11)
                // do not support ASSEMBLYPROPERTY
                if (!s_assemblyPropertyUnsupportedEngines.Contains(engineEdition))
                {
                    await udtCollection.GetMetadata(context, result).ConfigureAwait(false);
                }
            }

            // 3. Add TVPs from the server if supported
            MetaDataCollectionBase? tvpCollection = FindMetaDataCollection("_TVPs", context);
            if (tvpCollection != null)
            {
                await tvpCollection.GetMetadata(context, result).ConfigureAwait(false);
            }

How it could look:

            // 2. Add UDTs from the server
            const string GetEngineEditionSqlCommand = "SELECT SERVERPROPERTY('EngineEdition');";
            using SqlCommand command = ((SqlConnection)context.Connection).CreateCommand();

            command.CommandText = GetEngineEditionSqlCommand;
            int engineEdition = (int)(context.IsAsync ? await command.ExecuteScalarAsync(context.CancellationToken).ConfigureAwait(false) : command.ExecuteScalar());
            // Azure SQL Edge (9) throws an exception when querying sys.assemblies
            // Azure Synapse Analytics (6) and Azure Synapse serverless SQL pool (11)
            // do not support ASSEMBLYPROPERTY
            if (!s_assemblyPropertyUnsupportedEngines.Contains(engineEdition))
            {
                //await udtCollection.GetMetadata(context, result).ConfigureAwait(false);
                var udtCommand = new SqlCommandCollection("_UDTs", 2,
                    [new SupportedQuery("select types.assembly_class COLLATE database_default + ', ' + assemblies.name + ', Version=' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionMajor')) + '.' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionMinor')) + '.' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionBuild')) + '.' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'VersionRevision')) + ISNULL(', Culture=' + CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'CultureInfo')),'') + ISNULL(', PublicKeyToken=' + LOWER(REPLACE(CONVERT(VARCHAR(200),ASSEMBLYPROPERTY(assemblies.name, 'PublicKey'),1),'0x','')),'') TypeName, 29 ProviderDbType, max_length ColumnSize, null CreateFormat, null CreateParameters, null DataType, null IsAutoincrementable, null IsBestMatch, null IsCaseSensitive, is_fixed_length IsFixedLength, null IsFixedPrecisionScale, null IsLong, is_nullable IsNullable, 1 IsSearchable, null IsSearchableWithLike, null IsUnsigned, null MaximumScale, null MinimumScale, null IsConcurrencyType, 0 IsLiteralSupported, null LiteralPrefix, null LiteralSuffix, null NativeDataType from sys.assemblies as assemblies  join sys.assembly_types as types on assemblies.assembly_id = types.assembly_id")],
                    []);
                await udtCommand.GetMetadata(context, result).ConfigureAwait(false);
            }

            // 3. Add TVPs from the server
            var tvpCommand = new SqlCommandCollection("_TVPs", 2,
                    [new SupportedQuery("select name TypeName, 30 ProviderDbType, max_length ColumnSize, null CreateFormat, null CreateParameters, null DataType, null IsAutoincrementable, null IsBestMatch, null IsCaseSensitive, null IsFixedLength, null IsFixedPrecisionScale, null IsLong, is_nullable IsNullable, 0 IsSearchable, null IsSearchableWithLike, null IsUnsigned, null MaximumScale, null MinimumScale, null IsConcurrencyType, 0 IsLiteralSupported, null LiteralPrefix, null LiteralSuffix, null NativeDataType from sys.types  where is_table_type = 1")],
                    []);
            await tvpCommand.GetMetadata(context, result).ConfigureAwait(false);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also looking at the UDTs reading query, I'm thinking can we embed check for the SERVERPROPERTY('EngineEdition') into that query WHERE clause and not to check for it in separate query?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed here too. The list of schema collections is defined here and here, so I would personally expect that to remain constant for a given provider; happy for the SqlClient team to make that decision.

More concretely about the _UDTs and _TVPs collections, my question is: we know that _TVPs will always be supported, and the EngineEdition support guard is performed within the DataTypes collection evaluation. Do we gain anything by having the DataTypes collection look up these other collections, or can it simply execute the underlying UDT/TVP queries itself?

wrt putting the EngineEdition guard in the query: I'd advise against it. The query has to be compiled by SQL Server before it's run, and if it's run on a version of SQL Server which doesn't support sys.assemblies/ASSEMBLYPROPERTY then that compilation will throw an exception. Caching the engine edition in ConnectionCapabilities might be an interesting idea, but probably out of scope for this PR.

@tetolv tetolv Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now I have removed all support for the versionable collections. Versioning has remained only for datatypes, where json and vector actually depends on server negotiated capabilities. It really allowed to trim down a lot of code, but on the other hand, if we would need to reintroduce collection versioning, then we will need to reimplement it from scratch.

In a last commit I have also added an example of how I was wanted to push server edition check into UDT query, without breaking it on unsupported editions. I have used nested sp_executesql, which delays inner query compilation, until corresponding check for a valid server edition have succeeded. Will that solution be acceptable?

Also I would rather leave pattern:

var tvpCommand = new SqlCommandCollection("<collection_name>", "<query>", [<parameters>]);
await tvpCommand.GetMetadata(context, result).ConfigureAwait(false);

for reading UDTs and TVPs, because it takes on itself all the hassle of preparing, executing a query and then mapping all the results to a DataTable. Downside is that we still constructing a new instance of SqlCommandCollection, providing to it now unnecessary collection name, number of identity parts and restrictions. But still it is much shorter, than if we would handle all query handling manually.

(query.MinimumVersion == null || string.Compare(context.Caps.ServerVersion, query.MinimumVersion, StringComparison.OrdinalIgnoreCase) >= 0) &&
(query.MaximumVersion == null || string.Compare(context.Caps.ServerVersion, query.MaximumVersion, StringComparison.OrdinalIgnoreCase) <= 0))
{
Debug.Assert(validQuery == null, $"Two queries matches current version {context.Caps.ServerVersion} in collection {CollectionName}");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we keep the functionality: rather than assert, let's just break after the first match.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That also depends on the decision, if we should ditch support for multiple queries for different server versions. If we would support only one query per collection, then of course there we can break after the first matching collection is found, but if not, then assert would signal, if anyone would define incorrect collection filter conditions, so that two collection queries are returned simultaneously.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's only one query per collection

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now I have remove support for collection query versioning.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Localization Needed PRs that contain changes that would need localization

Projects

Status: Waiting for customer

Development

Successfully merging this pull request may close these issues.

8 participants