PSDataverse 2: Rebuilding a Fast Dataverse Tool for the Work That Actually Matters

Data migration rarely begins with an elegant architecture diagram.

It usually begins with a CSV file, an old database, an API that was never designed for bulk export, or two Dataverse environments that have quietly drifted apart. Somewhere between the source and destination, data must be cleaned, transformed, related, validated, retried, and—eventually—trusted.

That is the kind of work PSDataverse was created for.

PowerShell is remarkably good at connecting systems that were never designed to work together. It can read from SQL, CSV, JSON, Excel, REST APIs, legacy applications, or practically anything else, then transform that data and send it somewhere new. For Dataverse projects, this makes PowerShell much more than an administrative shell. It becomes a practical migration and synchronization platform.

PSDataverse brings the Dataverse Web API into that environment without trying to hide PowerShell behind another abstraction. You can pipe records, inspect responses, compose commands, and still reach the underlying Web API whenever a higher-level command is not enough.

Over the years, it has been used in real production work. That history gave me confidence in the module—but it also made me cautious about changing it.

When working code is no longer enough

Software that survives long enough accumulates assumptions.

Some are visible: old runtime targets, dated authentication flows, or commands that no longer feel natural. Others are buried much deeper: how connections are stored, when tokens are refreshed, whether cancellation reaches an HTTP request, what happens when one operation inside a batch fails, or whether a retry might accidentally repeat a write whose result is unknown.

PSDataverse was fast, useful, and proven. But parts of its internal design belonged to an earlier generation of .NET and PowerShell.

I could have applied a few isolated updates, changed the version number, and called it modernization. Instead, I decided to examine the module almost from first principles:

• What does a PowerShell developer expect a connection to look like?
• Which authentication methods belong in a modern Dataverse tool?
• Can parallelism remain fast without becoming uncontrolled?
• How should failures be presented when hundreds or thousands of records are involved?
• When is an individual request faster than $batch?
• When do Dataverse’s CreateMultiple and UpdateMultiple messages win?
• Which old behaviors should remain compatible, and which should be corrected?
• Can the package users install be proven to be the same package we tested?

Those questions became PSDataverse 2.

Performance is a feature, but predictability is part of performance

Performance has always been one of PSDataverse’s defining qualities. I wanted to preserve that, but I did not want to optimize based on assumptions.

It is easy to run fifty GET requests, observe that more parallelism is faster, and generalize that conclusion to every workload. Dataverse writes are more complicated. POST, PATCH, and DELETE have different characteristics. Individual requests, $batch, and the multiple-operation messages have different transactional, ordering, and failure semantics.

So I built a guarded benchmark that creates a temporary Dataverse table, fills it with data, updates it, verifies the results, deletes the records, and finally removes the table. It measures individual requests, batch envelopes, and the bulk-style CreateMultiple and UpdateMultiple messages independently.

Then I repeated the tests with warm-up operations, randomized scenario order, multiple samples, different batch sizes, and concurrent envelopes.

The results reinforced something important: there is no universally fastest transport.

In one measured workload, five concurrent batch envelopes reached a median of approximately 165 POST operations per second, compared with about 58 for individual requests at a degree of parallelism of 20. For PATCH, two concurrent UpdateMultiple envelopes reached approximately 132 operations per second, compared with about 51 individual operations per second.

DELETE behaved differently again. Beyond a certain level, more parallelism delivered little additional benefit.

Those numbers are not promises. Plug-ins, payload size, relationships, alternate keys, network conditions, and Dataverse service-protection limits can change the result dramatically. Their value is in demonstrating why PSDataverse should give developers explicit, well-designed choices instead of silently deciding that one transport is always best.

The new request engine is built around bounded channels. It applies backpressure instead of continually creating more work, carries cancellation through requests and retry delays, honors Dataverse concurrency hints, and respects Retry-Afterguidance.

Just as importantly, it avoids automatically replaying ambiguous writes. Retrying a GET is usually harmless. Repeating a POST after losing the response may create duplicate data. Performance should never come from quietly gambling with correctness.

Connections needed to become real objects

The old model treated a connection too much like shared session state. That can be convenient for the first command, but it becomes fragile when a script talks to multiple environments.

Environment synchronization makes this particularly obvious. A script may need to read from development, compare against test, and write into another environment. Authentication tokens, URLs, concurrency hints, and cached capabilities must remain isolated.

PSDataverse 2 introduces first-class connection objects and a session-scoped connection registry. Connections can be named, inspected, selected explicitly, made the default, refreshed, and disposed deterministically.

This makes simple scripts remain simple while allowing larger migration tools to be precise:

$source = Connect-Dataverse $sourceUrl -Interactive -ConnectionName Source
$target = Connect-Dataverse $targetUrl -Interactive -ConnectionName Target

Get-DataverseRow -TableName accounts -Connection $source |
    Set-DataverseRow -TableName accounts -Connection $target

Real migrations need more mapping and validation than this example shows, but the important idea is visible: the source and target are explicit. A token, URL, or HTTP client cannot accidentally leak from one environment into the other.

Authentication should match how people work today

Authentication has changed significantly since the first PSDataverse versions.

Interactive authentication on Windows now uses Web Account Manager by default, giving users the account selection and passwordless experiences provided by Windows and Microsoft Entra. Other platforms use the system browser.

Device code remains useful for terminals and remote scenarios. Application authentication supports client secrets and certificates. Existing access tokens and custom token providers are supported for integration into larger systems. Integrated Windows Authentication remains available for compatible federated identities, even though it is no longer the right choice for most managed Entra users.

Passkeys do not need to become a special PSDataverse authentication mechanism. They appear naturally through the modern interactive identity experience.

I also kept compatible connection strings as a migration path. A new major version can improve its design without making existing users feel abandoned.

Better commands for both small scripts and large migrations

A low-level Web API command is essential because Dataverse is broad and constantly evolving. PSDataverse continues to provide that escape hatch through Invoke-DataverseRequest, with Send-DataverseOperation retained as a compatibility alias.

But developers should not have to manually construct every common request.

PSDataverse 2 adds PowerShell-oriented commands for:

• Row creation, retrieval, update, and deletion
• Table metadata and provisioning
• Actions and functions
• CSV and JSON import and export
• Connection inspection and testing
• CreateMultiple, UpdateMultiple, and UpsertMultiple workloads
• Detection of whether a table supports those multiple-operation messages

The bulk-capability check is cached on each connection so it does not become a new performance tax. Unsupported workloads fail before thousands of records are sent.

Error reporting has also been redesigned for large operations. When a multiple-operation chunk fails, PSDataverse can identify the failed chunk, its source row range, retained input objects, failed content IDs, and successful sibling chunks. A migration script can therefore report or retry the affected records without forcing someone to reverse-engineer a large response.

Batch responses receive the same attention. Empty bodies, malformed proxy responses, duplicate headers, throttling responses, mixed line endings, and failed content IDs are handled deterministically.

When moving important data, “something failed somewhere in the batch” is not an acceptable diagnostic.

Why the testing took so long

This release touches authentication, concurrency, packaging, public commands, error handling, and compatibility. Each area can look correct in isolation while failing when combined with another.

That is why I spent so much time testing scenarios rather than only testing methods.

The test suite covers cancellation, retries, pagination, authentication parameter sets, token refresh coordination, cross-environment isolation, request ordering, batch parsing, malformed responses, bulk capability detection, and structured failures.

A guarded live integration suite creates a disposable Dataverse table and exercises CRUD, pagination, concurrent batches, CreateMultiple, UpdateMultiple, UpsertMultiple, negative cases, and cleanup against a real environment.

The release package is built twice from clean directories. The resulting module inventories and canonical package hashes must match. That exact package is then installed through a temporary local PowerShell repository and tested outside the source tree.

Finally, the same package is installed and validated on Windows, Linux, and macOS using PowerShell 7.6.

The goal is not to claim that bugs are now impossible. The goal is to make every release claim traceable to a test that meaningfully represents how the module is used.

A deliberate new foundation

PSDataverse 2 targets .NET 10 and requires PowerShell 7.6. PowerShell 5.1 and older PowerShell 7 releases cannot load this new binary architecture.

That is a significant compatibility decision, and I did not make it casually.

Trying to modernize authentication, concurrency, dependency management, and runtime behavior while preserving every historical runtime would have forced the new design to carry too many old constraints. Instead, PSDataverse 2 provides a migration guide, retains familiar concepts where they remain sound, and keeps inexpensive compatibility features such as the Send-DataverseOperation alias.

Unused Scriban and Humanizer functionality has been removed. Compiler and analyzer warnings have been addressed. The remaining obsolete Integrated Windows Authentication call is retained intentionally and isolated because some organizations still need it.

This is not change for its own sake. It is an attempt to give PSDataverse another strong foundation for years of real-world use.

Why PSDataverse still matters

Data integration projects exist at every scale.

A small project might need to import a few hundred reference records from CSV. A larger migration may move millions of rows from SQL, preserve alternate keys, establish relationships in later phases, and record every rejected item for reconciliation. An environment synchronization process may continuously compare configuration data across development, test, and production.

Dedicated migration platforms can be valuable, but they are not always available, affordable, or flexible enough. Sometimes the source is unusual. Sometimes the transformation rules are specific to one project. Sometimes a developer needs a transparent tool they can inspect, automate, and adapt immediately.

PowerShell is often the shortest distance between those systems.

PSDataverse matters because it lets PowerShell developers work with Dataverse at that level without giving up performance, control, or visibility. It can be the simple command in a small script and the request engine underneath a serious migration framework.

That combination—approachable at the beginning, explicit when things become complicated—is what I wanted to protect while rebuilding it.

Try the release candidate

PSDataverse 2.0.0-rc1 is now available as a public release candidate and requires PowerShell 7.6 or later:

Install-PSResource -Name PSDataverse -Version 2.0.0-rc1 -Prerelease

PowerShellGet users can install it with:

Install-Module -Name PSDataverse -RequiredVersion 2.0.0-rc1 -AllowPrerelease

If you use Dataverse for migrations, synchronization, deployment preparation, or everyday automation, I would be grateful for your feedback—especially from workloads and environments that differ from the ones I tested.

PSDataverse 2 represents a great deal of design, rewriting, benchmarking, and careful validation. More importantly, it reflects years of lessons from using PowerShell and Dataverse together.

The release candidate is an invitation to help shape what comes next.