If I have a JSON object that has many fields > 3000 and with some fields that have nested objects, is there a limit to how many columns/fields you can have?
Writing the ingestion schema will take time but is it worth a try with that many fields/columns?
Honestly not tested.
Is it possible for you to write ingestion spec without specifying anything in the dimension list so that it will take all of them?
If you explicitly specify 300 out of 3000, then it will take only those 300 but if you leave dimension list empty, then it will take all.
That way , at least you don’t need to manually write all 3000.
Good idea, will try it!