Document Intelligence Infrastructure

Article covers GraphRAG and an agentic approach as a path to document intelligence,

Process Based Automation

If you have a strong, underlying process, it should be trivial to automate it.

The Mesh Network

A mesh network is a type of LAN topology where every node connects (in a flat layout) to as many other nodes as possible. Nodes can then work together to route data as efficiently as possible.

Big Data's New Vacation Home - The Lakehouse; Microsoft's Approach

The lakehouse concept combines the capabilities of data lakes (which have scalability qualities) and data warehouses (which have advanced query functionality).

Microsoft Fabric's resources on Data Engineering delves into the concept of data lakehouse (and Microsoft's SaaS implementation, OneLake, billed as OneDrive for data)  with notes on how the lakehouse makes use of Apache Spark.

Public APIs and Democratized Data

A directory can be found here.

Analytics Libraries Expect Regularised Data

This is a recurrent theme in quantitative computing. 

Analytics libraries expect clean, regularised data, e.g. time series with no missing values.  Real-life data often has gaps and idiosyncrasies - it needs to cleaned often (to create a golden source) but even then subsequently rejigged based on the consumer need. 

This is akin to the Adapter Design Pattern in programming. In the Adapter you adapt an "interface" to another "interface" - for example, an XML dataset is "adapted" into a JSON dataset for JSON consumers.

Statistical libraries in particular are particularly picky about datasets and consistency, particularly when comparing datasets and trying to find relationships or errors between actual and expected values.

LM Studio Setup & The "Logged In" User

LM Studio has two setup options:

1. For anyone who uses the computer

2. Only for the currently logged in user

Both have advantages, but if you are working with Studio to build custom LLMs tailored to you, you may want option 2 despite the (potential) convenience of option 1 where LM Studio is "universally" available to all users of the machine.

Data Flywheels

 The concept of a data flywheel is central to continuous improvement of AI systems.

Data Lineage

The modern day Windows maven cannot be solely excellent at all the latest characteristics of the Dot Net platform but needs a broader understanding of wider computing issues.

One such issue is data lineage, a subcategory of the field of data governance. Data lineage relates to understanding the flow of data from source to destination, including any transformations that occur along the way.

Roll your own MD5 implementation - here's how!

Roll your own MD5 implementation - here's how!

If you wish to implement an MD5 algorithm (commonly used to verify data integrity) in C# the "right way" to go about it is to inherit from class MD5.

public (abstract) class MD5: HashAlgorithm  [From mscorlib and in System.Security.Cryptography]

public (abstract) class HashAlgorithm: IDisposable, ICryptoTransform

Brainchild of Yale Mathematician Ron Rivest!

The MD5 ("message digest") algorithm was designed in 1991 by Ronald Rivest (whose academic journey began with a Bachelors in Mathematics from Yale) as the successor to MD4, although execution speed is slower. It is described in RFC1321. It produces a 128 bit digest or "fingerprint" of the input data. That "fingerprint" can then be used to validate the integrity of downloaded data files.

The Avalanche Effect

A concept that comes up in discussions of MD5 and other related algorithms is the notion of the avalanche effect. This is a scenario where an input bit is allowed to impact two or more output bits. Berlin-born cryptographer Horst Feistel, whose research led to the formation of the Data Encryption Standard, or DES, in the 1970s, and gave his name to the Feistel cipher, coined the term.

The Idea of a One Way Hash Function or Cryptographic Checksum

To understand MD5 as a cryptographic checksum or one-way hash function, we should understand the basics of hash functions.  A hash function converts a variable length input into a fixed length output (the former is known as the "pre-image" and the latter the "hash value"). The application is verification - to validate if a given "pre-image" is "likely" to be the same as the real "pre-image" the user is trying to validate.  The key to a cryptographic hash, or one-way hash, is that it is difficult to create a "pre-image" that hashes to a particular value. In short, it is hard to spoof the true "pre-image".