Understanding Neural Networks

Understanding Neural Networks

by | Dec 7, 2023 | A.I

Neural Networks: An Overview

Neural networks are a cornerstone of artificial intelligence (AI), simulating the way human brains analyze and process information. They consist of interconnected nodes, mirroring the structure of neurons in the brain, and are employed to recognize patterns and solve complex problems in various fields including speech recognition, image processing, and data analysis.

Introduction to Neural Networks

Neural networks are computational models inspired by the human brain’s interconnected neuron structure. They are part of a broader field called machine learning, where algorithms learn from and make predictions or decisions based on data. The basic building block of a neural network is the neuron, also known as a node or perceptron. These neurons are arranged in layers: an input layer to receive the data, hidden layers to process it, and an output layer to produce the final result. Each neuron in one layer is connected to neurons in the next layer, and these connections have associated weights that adjust as the network learns from data.

Brief History

The concept of neural networks dates back to the 1940s when Warren McCulloch and Walter Pitts created a computational model for neural networks. In 1958, Frank Rosenblatt invented the perceptron, an algorithm for pattern recognition based on a two-layer learning computer network. However, the interest in neural networks declined in the 1960s due to limitations in computing power and theoretical understanding.

The resurgence of interest in neural networks occurred in the 1980s, thanks to the backpropagation algorithm, which effectively trained multi-layer networks, and the increase in computational power. This resurgence continued into the 21st century with the advent of deep learning, where neural networks with many layers (deep neural networks) achieved remarkable success in various fields.

A Simple Example

Consider a simple neural network used for classifying emails as either ‘spam’ or ‘not spam.’ The input layer receives features of the emails, such as frequency of certain words, email length, and sender’s address. The hidden layers process these inputs by performing weighted calculations, passing the results from one layer to the next. The final output layer categorizes the email based on the processed information, using a function that decides whether it’s more likely to be ‘spam’ or ‘not spam.’

Conclusion

Neural networks, with their ability to learn from data and make complex decisions, have become integral to advancements in AI. As computational power and data availability continue to increase, neural networks are poised to drive significant innovations across various sectors.


Decision Trees and Naive Bayes Classifiers

Decision Trees and Naive Bayes Classifiers

by | Dec 6, 2023 | A.I

Decision Trees and Naive Bayes Classifiers

Decision Trees

Overview:

  • Decision trees are a type of supervised learning algorithm used for classification and regression tasks.
  • They work by breaking down a dataset into smaller subsets while at the same time developing an associated decision tree incrementally.
  • The final model is a tree with decision nodes and leaf nodes. A decision node has two or more branches, and a leaf node represents a classification or decision.

Brief History:

  • The concept of decision trees can be traced back to the work of R.A. Fisher in the 1930s, but modern decision tree algorithms emerged in the 1960s and 1970s.
  • One of the earliest and most famous decision tree algorithms, ID3 (Iterative Dichotomiser 3), was developed by Ross Quinlan in the 1980s.
  • Subsequently, Quinlan developed the C4.5 algorithm, which became a standard in the field.

Simple Example:

Imagine a decision tree used to decide if one should play tennis based on weather conditions. The tree might have decision nodes like ‘Is it raining?’ or ‘Is the humidity high?’ leading to outcomes like ‘Play’ or ‘Don’t Play’.

Naive Bayes Classifiers

Overview:

  • Naive Bayes classifiers are a family of simple probabilistic classifiers based on applying Bayes’ theorem with strong independence assumptions between the features.
  • They are highly scalable and can handle a large number of features, making them suitable for text classification, spam filtering, and even medical diagnosis.

Brief History:

  • The foundation of Naive Bayes is Bayes’ theorem, formulated by Thomas Bayes in the 18th century.
  • However, the ‘naive’ version, assuming feature independence, was developed and gained prominence in the 20th century, particularly in the 1950s and 1960s.
  • Naive Bayes has remained popular due to its simplicity, effectiveness, and efficiency.

Simple Example:

Consider a Naive Bayes classifier for spam detection. It calculates the probability of an email being spam based on the frequency of words typically found in spam emails, such as “prize,” “free,” or “winner.”

Conclusion

Both decision trees and Naive Bayes classifiers are instrumental in the field of machine learning, each with its strengths and weaknesses. Decision trees are known for their interpretability and simplicity, while Naive Bayes classifiers are appreciated for their efficiency and performance in high-dimensional spaces. Their development and application over the years have significantly contributed to the advancement of machine learning and data science.


Machine Learning: History, Concepts, and Application

Machine Learning: History, Concepts, and Application

by | Dec 5, 2023 | A.I

Brief History and Early Use Cases of Machine Learning

Machine learning began shaping in the mid-20th century, with Alan Turing’s 1950 paper “Computing Machinery and Intelligence” introducing the concept of machines learning like humans. This period marked the start of algorithms based on statistical methods.

The first documented attempts at machine learning focused on pattern recognition and basic learning algorithms. In the 1950s and 1960s, early models like the perceptron emerged, capable of simple learning tasks such as visual pattern differentiation.

Three Early Use Cases of Machine Learning:

  1. Checker-Playing Program: One of the earliest practical applications was in the late 1950s when Arthur Samuel developed a program that could play checkers, improving its performance over time by learning from each game.
  2. Speech Recognition: In the 1970s, Carnegie Mellon University developed “Harpy,” a speech recognition system that could comprehend approximately 1,000 words, showcasing early success in machine learning for speech recognition.
  3. Optical Character Recognition (OCR): Early OCR systems in the 1970s and 1980s used machine learning to recognize text and characters in images, a significant advancement for digital document processing and automation.

How Machine Learning Works

Data Collection: The process starts with the collection of diverse data.

Data Preparation: This data is cleaned and formatted for use in algorithms.

Choosing a Model: A model like decision trees or neural networks is chosen based on the problem.

Training the Model: The model is trained with a portion of the data to learn patterns.

Evaluation: The model is evaluated using a separate dataset to test its effectiveness.

Parameter Tuning: The model is adjusted to improve its performance.

Prediction or Decision Making: The trained model is then used for predictions or decision-making.

A Simple Example: Email Spam Detection

Let’s consider an email spam detection system as an example of machine learning in action:

  1. Data Collection: Emails are collected and labeled as “spam” or “not spam.”
  2. Data Preparation: Features such as word presence and email length are extracted.
  3. Choosing a Model: A decision tree or Naive Bayes classifier is selected.
  4. Training the Model: The model learns to associate features with spam or non-spam.
  5. Evaluation: The model’s accuracy is assessed on a different set of emails.
  6. Parameter Tuning: The model is fine-tuned for improved performance.
  7. Prediction: Finally, the model is used to identify spam in new emails.

Conclusion

Machine learning, from its theoretical inception to its contemporary applications, has undergone significant evolution. It encompasses the preparation of data, selection and training of a model, and the utilization of that model for prediction or decision-making. The example of email spam detection is just one of the many practical applications of machine learning that impact our daily lives.

 

Divide and Conquer: Subtle Strategies for Supercharging Your Database Performance

Divide and Conquer: Subtle Strategies for Supercharging Your Database Performance

by | Dec 4, 2023 | Database

Database Table Partitioning

Database table partitioning is a strategy used to divide a large database table into smaller, manageable segments, known as partitions, while maintaining the overall structure and functionality of the table. This technique is implemented in database management systems like Microsoft SQL Server (MSSQL) and PostgreSQL (Postgres).

What is Database Table Partitioning?

Database table partitioning involves breaking down a large table into smaller segments. Each partition contains a subset of the table’s data, based on specific criteria such as date ranges or geographic locations. This allows for more efficient data management and can significantly improve performance for certain types of queries.

Impact of Partitioning on CRUD Operations

  • Create: Streamlines the insertion of new records to the appropriate partition, leading to faster insert operations.
  • Read: Enhances query performance as searches can be limited to relevant partitions, accelerating read operations.
  • Update: Makes updating data more efficient, but may add overhead if data moves across partitions.
  • Delete: Simplifies and speeds up deletion, especially when dropping entire partitions.

Advantages of Database Table Partitioning

  • Improved Performance: Particularly for read operations, partitioning can significantly enhance query speeds.
  • Easier Data Management: Managing smaller partitions is more straightforward.
  • Efficient Maintenance: Maintenance tasks can be conducted on individual partitions.
  • Organized Data Structure: Helps in logically organizing data.

Disadvantages of Database Table Partitioning

  • Increased Complexity: Adds complexity to database management.
  • Resource Overhead: May require more disk space and memory.
  • Uneven Performance Risks: Incorrect partition sizing or data distribution can lead to bottlenecks.

MSSQL Server: Example Scenario

In MSSQL, table partitioning involves partition functions and schemes. For example, a SalesData table can be partitioned by year, enhancing CRUD operation efficiency. Here’s an example of how you might partition a table in MSSQL:

-- Create a partition function
CREATE PARTITION FUNCTION SalesDataYearPF (int)
AS RANGE RIGHT FOR VALUES (2015, 2016, 2017, 2018, 2019, 2020);

-- Create a partition scheme
CREATE PARTITION SCHEME SalesDataYearPS
AS PARTITION SalesDataYearPF ALL TO ([PRIMARY]);

-- Create a partitioned table
CREATE TABLE SalesData
(
    SalesID int IDENTITY(1,1) NOT NULL,
    SalesYear int NOT NULL,
    SalesAmount decimal(10,2) NOT NULL
) ON SalesDataYearPS (SalesYear);

PostgreSQL: Example Scenario

In Postgres, partitioning uses table inheritance. A rapidly growing Logs table can be partitioned monthly, optimizing CRUD operations. Here’s an example of how you might partition a table in PostgreSQL:

-- Create a master table
CREATE TABLE logs (
    logdate DATE NOT NULL,
    logevent TEXT
) PARTITION BY RANGE (logdate);

-- Create partitions
CREATE TABLE logs_y2020m01 PARTITION OF logs
    FOR VALUES FROM ('2020-01-01') TO ('2020-02-01');

CREATE TABLE logs_y2020m02 PARTITION OF logs
    FOR VALUES FROM ('2020-02-01') TO ('2020-03-01');

Conclusion

Database table partitioning in MSSQL and Postgres significantly affects CRUD operations. While offering benefits like improved query speed and streamlined data management, it also introduces complexities and demands careful planning. By understanding the advantages and disadvantages of partitioning, and by using the appropriate SQL commands for your specific database system, you can effectively implement this powerful tool in your data management strategy.

 

Understanding AI, AGI, ML, and Language Models

Understanding AI, AGI, ML, and Language Models

by | Dec 4, 2023 | A.I

Understanding AI, AGI, ML, and Language Models

Artificial Intelligence (AI) is a broad field in computer science concerned with building smart machines capable of performing tasks that typically require human intelligence. AI encompasses various subfields, including machine learning, natural language processing, robotics, and more. Its primary goal is to enable computers to perform tasks such as decision-making, problem-solving, perception, and understanding human language.

Machine Learning (ML), a subset of AI, focuses on developing algorithms and statistical models that enable computers to learn from and make predictions or decisions based on data. Unlike traditional programming, where humans explicitly code the behavior, machine learning allows systems to automatically learn and improve from experience. This learning process is driven by feeding algorithms large amounts of data and allowing them to adjust and improve their performance over time.

One of the most notable applications of ML is in the development of Language Models (LMs), which are algorithms designed to understand, interpret, and generate human language. These models are trained on vast datasets of text and can perform a range of language-related tasks, such as translation, summarization, and even generating human-like text. Language models like GPT (Generative Pretrained Transformer) are examples of how AI and ML converge to create sophisticated tools for natural language processing.

Artificial General Intelligence (AGI), on the other hand, represents a level of AI that is far more advanced and versatile. While current AI systems, including language models, are designed for specific tasks (referred to as narrow AI), AGI refers to a hypothetical AI that has the ability to understand, learn, and apply its intelligence broadly and flexibly, much like a human. AGI would possess the ability to reason, solve problems, comprehend complex ideas, learn from experience, and apply its knowledge to a wide range of domains, effectively demonstrating human-like cognitive abilities.

The relationship between AI, ML, AGI, and language models is one of a nested hierarchy. AI is the broadest category, under which ML is a crucial methodology. Language models are specific applications within ML, showcasing its capabilities in understanding and generating human language. AGI, while still theoretical, represents the potential future of AI where systems could perform a wide range of cognitive tasks across different domains, transcending the capabilities of current narrow AI systems.

In summary, AI is a vast field aimed at creating intelligent machines, with machine learning being a key component that focuses on data-driven learning and adaptation. Language models are a product of advancements in ML, designed to handle complex language tasks. AGI remains a goal for the future, representing a stage where AI could match or surpass human cognitive abilities across a broad spectrum of tasks and domains.

 

PostgreSQL Full-text search using “text search vectors”

PostgreSQL Full-text search using “text search vectors”

by | Oct 5, 2023 | Postgres, Search

Postgres Vector Search

Full-text search in PostgreSQL is implemented using a concept called “text search vectors” (or just “tsvector”). Let’s dive into how it works:

  1. Text Search Vectors (tsvector):
    • A tsvector is a sorted list of distinct lexemes, which are words that have been normalized to merge different forms of the same word (e.g., “run” and “running”).
    • PostgreSQL provides functions to convert plain text into tsvector format, which typically involves:
      • Parsing the text into tokens.
      • Converting tokens to lexemes.
      • Removing stop words (common words like “and” or “the” that are typically ignored in searches).
    • Example: The text “The quick brown fox” might be represented in tsvector as ‘brown’:3 ‘fox’:4 ‘quick’:2.
  2. Text Search Queries (tsquery):
    • A tsquery represents a text search query, which includes lexemes and optional operators.
    • Operators can be used to combine lexemes in different ways (e.g., AND, OR, NOT).
    • Example: The query “quick & fox” would match any tsvector containing both “quick” and “fox”.
  3. Searching:
    • PostgreSQL provides the @@ operator to search a tsvector column with a tsquery.
    • Example: WHERE column @@ to_tsquery(‘english’, ‘quick & fox’).
  4. Ranking:
    • Once you’ve found matches using the @@ operator, you often want to rank them by relevance.
    • PostgreSQL provides the ts_rank function to rank results. It returns a number indicating how relevant a tsvector is to a tsquery.
    • The ranking is based on various factors, including the frequency of lexemes and their proximity to each other in the text.
  5. Indexes:
    • One of the significant advantages of tsvector is that you can create a GiST or GIN index on it.
    • These indexes significantly speed up full-text search queries.
    • GIN indexes, in particular, are optimized for tsvector and provide very fast lookups.
  6. Normalization and Configuration:
    • PostgreSQL supports multiple configurations (e.g., “english”, “french”) that determine how text is tokenized and which stop words are used.
    • This allows you to tailor your full-text search to specific languages or requirements.
  7. Highlighting and Snippets:
    • In addition to just searching, PostgreSQL provides functions like ts_headline to return snippets of the original text with search terms highlighted.

In summary, PostgreSQL’s full-text search works by converting regular text into a normalized format (tsvector) that is optimized for searching. This combined with powerful query capabilities (tsquery) and indexing options makes it a robust solution for many full-text search needs.

Implementing vector search using E.F Core and Postgres SQL

here are the steps to implement vector search in your dot net project:

Step 1: Add the required nuget packages

   
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="7.0.11" />
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL.Design" Version="1.1.0" />
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL.NetTopologySuite" Version="7.0.11" />

Step 2: Implement a vector in your entities by implementing properties of type NpgsqlTsVector as shown below

public class Blog
{
    public int Id { get; set; }
    public string Title { get; set; }
    public NpgsqlTsVector SearchVector { get; set; }
}

Step 3: add a computed column in your DbContext

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Blog>()
        .Property(b => b.SearchVector)
      .HasComputedColumnSql("to_tsvector('english', \"Blogs\".\"Title\")", stored: true);
}

in this case you are calculating the vector using the value of the title column of the blogs table, you can calculate the vector using a single column or a combination of columns

Now you are ready to use vector search in your queries, please check the example below

var searchTerm = "Jungle"; // Example search term
var searchVector = NpgsqlTsVector.Parse(searchTerm);

var blogs = context.Blogs
    .Where(p => p.SearchVector.Matches(searchTerm))
    .OrderByDescending(td => td.SearchVector.Rank(EF.Functions.ToTsQuery(searchTerm))).ToList();

In real world scenarios its better to create a vector by joining the values of several columns and weight them according to the relevance for your business case, you can check the test project I have created here : https://github.com/egarim/PostgresVectorSearch

and that’s it for this post, until next time, happy coding ))

 

 

 

 

Elastic Search

Elastic Search

by | Sep 23, 2023 | Search

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. It is a technology that is part of the Elastic Stack, along with Logstash, and Kibana, collectively known as the ELK Stack. Elasticsearch is built on top of the open-source Lucene library and provides a multi-tenant capable full-text search engine. It’s designed to be scalable, resilient, and very fast, which makes it a popular choice for many different types of applications including:

1. Search Engines: Full-text search, partial text search, faceted search, and more.

2. Log and Event Data Analysis: Often used with Logstash and Kibana for searching, analyzing, and visualizing log data in real-time.xx

3. Real-time Analytics: Can be used for analyzing large volumes of real-time data efficiently.

4. Data Visualization: Often used with Kibana to visualize the data stored in Elasticsearch.

5. Autocomplete Features: Quick search suggestions.

6. Geospatial Search: Searching based on geographic location.

Key Features:

  • Distributed and Scalable: Built to scale horizontally with easy distribution across multiple nodes.
  • Schema-free JSON Documents: Uses JSON documents in order to store data, which makes it flexible and easy to use.
  • RESTful API: Exposes REST APIs for CRUD operations, allowing interaction via standard HTTP methods.
  • Real-time Indexing: As soon as a document is stored, it is available for search.
  • Multi-tenancy: Supports multiple indices, and the indices can be divided into shards for better performance.

Basic Concepts:

  • Node: A single running instance of Elasticsearch.
  • Cluster: A collection of one or more nodes.
  • Index: A collection of documents having somewhat similar characteristics.
  • Shard: A subset of an index. Each shard is a self-contained index.
  • Replica: A copy of a shard for failover and increased performance.

Elasticsearch is widely used in a variety of applications that require complex search features, large-scale logging, or real-time analytics. It’s often compared to other NoSQL databases like MongoDB and Apache Solr.

Choosing to use Elasticsearch depends on your specific needs, but there are several compelling reasons why it might be a good fit for your project:

Speed

  • Fast Search: Built on top of Apache Lucene, Elasticsearch is designed for fast, real-time search operations.
  • Real-time Indexing: New data is searchable almost immediately after it’s added.

Scalability

  • Horizontal Scaling: You can easily add more nodes to your Elasticsearch cluster as your data and query volume grow.
  • Distributed Nature: Automatically distributes data and query load across all the available nodes in the cluster.

Flexibility

  •  Schema-less: You can index JSON documents without a predefined schema.
  •  RESTful API: Easily interact with the search engine through RESTful APIs, using JSON over HTTP.
  •  Multiple Data Types: Supports text, numbers, dates, geospatial data, and more.

Robustness

  • High Availability: Multiple copies of data (replicas) can be maintained to provide failover.
  • Built-in Cluster Health and Monitoring: Tools like Kibana can provide insights into the operations and health of your Elasticsearch cluster.

Rich Query DSL

  • Powerful Query Language: Elasticsearch provides a rich, flexible, query language (DSL) that can perform complex queries, filters, and aggregations.
  • Relevancy Scoring: Sophisticated algorithms score each document for its relevance to a given search query.

Integration and Extensibility

  • Part of the Elastic Stack: Integrates seamlessly with other components like Logstash for data ingestion and Kibana for data visualization.
  • Extensible: Supports plugins to add additional features and capabilities.

Multi-Tenancy

  • Support for Multiple Indices: You can have multiple indices (databases) and query them all at once if needed.

Use Cases

  • Full-text Search: For applications like e-commerce product search, media catalog search, etc.
  • Logging and Log Analysis: When combined with Logstash and Kibana, it’s a powerful tool for logging debug information, monitoring, and real-time analytics.
  • Real-time Analytics: For business intelligence, performance metrics, and other real-time analytics needs.
  • Data Visualization: Can be used with Kibana or other visualization tools to graphically represent your data.

Community and Ecosystem

  • Strong Community: A large, active community contributes to its robust set of features.
  • Comprehensive Documentation: Extensive online resources are available to help you get the most out of Elasticsearch.

 

However, it’s important to note that Elasticsearch may not be suitable for all types of projects. It can be resource-intensive, and the learning curve can be steep if you’re new to search and analytics engines. It might also be overkill for simple search needs or small datasets. Always consider your specific requirements and constraints when deciding whether to use Elasticsearch.

 

 

Interplanetary File System (IPFS)

Interplanetary File System (IPFS)

by | Sep 15, 2023 | IPFS

IPFS stands for the InterPlanetary File System. It is a protocol and network designed to create a peer-to-peer method of storing and sharing hypermedia in a distributed file system. IPFS was initially designed by Juan Benet and is now an open-source project with a large community of contributors.

How IPFS Works

In a traditional client-server model like HTTP, your computer (the client) requests information from a specific server. This creates a centralized point of failure; if the server goes down or is slow, you can’t access your information.

IPFS aims to decentralize the web by creating a peer-to-peer network where each computer can host files, or parts of files, making the network more robust and potentially faster. Here’s a simplified explanation of how IPFS works:

1. Content Addressing: Unlike traditional file systems that locate data based on where it is stored (file location), IPFS locates files based on what they are (file content). Each file and all of the blocks within it are given a unique fingerprint called a cryptographic hash.

2. Distributed Storage: Files are split into blocks, and each block is stored across a decentralized network of nodes. When you look up a file, you’re asking the network to find nodes that are storing the blocks that make up the file.

3. Data Retrieval: When you want to access a file, your computer asks the network for the blocks that make up the file. It can then reassemble the file for use. This can happen much faster as multiple nodes might be closer to you or have parts of the file, allowing for parallel downloads.

4. Immutable and Versioned: Files are immutable, meaning they can’t be changed without altering the hash of the file. This also means that every version of every file is permanently stored. This is advantageous for archiving and versioning but can be a challenge for mutable data.

5. Node Involvement: Anyone can operate a node, and by doing so, contribute to storing and distributing content. Nodes can also cache popular content to improve data retrieval speed and reduce the burden on individual nodes.

Advantages of IPFS

  • Decentralization: Removes single points of failure in the network.
  • Performance: Potentially faster than traditional systems because data can be distributed more efficiently.
  • Censorship Resistance: Harder to censor or control content.
  • Permanent Web: Content-addressing allows for a more robust and permanent web.

Disadvantages of IPFS

  • Complexity: The architecture is more complex than traditional client-server models.
  • Data Redundancy: Every version of every file being stored can consume a lot of storage.
  • Legal and Ethical Issues: As with any file-sharing system, there’s the potential for misuse, such as sharing copyrighted or illegal material.

IPFS has gained attention and usage in various sectors including web development, scientific data, and blockchain. It’s often mentioned in the same breath as other decentralized technologies like blockchain.

Blazor WebAssembly and SQLite: Unleashing the Full Potential of Client-Side Data

Blazor WebAssembly and SQLite: Unleashing the Full Potential of Client-Side Data

by | Jul 26, 2023 | Blazor, C#, Sqlite, WebAssembly

In the evolving panorama of contemporary web application development, a technology that has particularly caught attention is Microsoft’s Blazor WebAssembly. This powerful tool allows for a transformative approach to managing and interacting with client-side data, offering innovative capabilities that are shaping the future of web applications.

Understanding Blazor WebAssembly

 

Blazor WebAssembly is a client-side web framework from Microsoft. It allows developers to build interactive web applications using C# instead of JavaScript. As the name suggests, it uses WebAssembly, a binary instruction format for a stack-based virtual machine, providing developers with the ability to run client-side web applications directly in the browser using .NET.

 

The Power of SQLite

 

SQLite, on the other hand, is a software library that provides a relational database management system (RDBMS). It operates directly on disk files without the need for a separate server process, making it ideal for applications that need local storage. It’s compact, requires zero-configuration, and supports most of the SQL standard, making it an excellent choice for client-side data storage and manipulation.

 

Combining Blazor WebAssembly with SQLite

 

By combining these two technologies, you can unlock the full potential of client-side data handling. Here’s how:

 

Self-Contained and Cross-Platform Development

 

Both Blazor WebAssembly and SQLite are self-contained systems, requiring no external dependencies. They also both provide excellent cross-platform support. This makes your applications highly portable and reduces the complexity of the development environment.

Offline Availability

 

SQLite enables the storage of data directly in the client’s browser, allowing your Blazor applications to work offline. Changes made offline can be synced with the server database once the application goes back online, providing a seamless user experience.

 

Superior Performance

 

Blazor WebAssembly runs directly in the browser, offering near-native performance. SQLite, being a lightweight yet powerful database, reads and writes directly to ordinary disk files, providing high-speed data access. This combination ensures your application runs quickly and smoothly.

 

Full .NET Support and Shared Codebase

With Blazor, you can use .NET for both client and server-side code, enabling code sharing and eliminating the need to switch between languages. Coupled with SQLite, developers can use Entity Framework Core to interact with the database, maintaining a consistent, .NET-centric development experience.

 

Where does the magic happens?

 

The functionality of SQLite with WebAssembly may vary based on your target framework. If you’re utilizing .NET 6 and Microsoft.Data.SQLite 6, your code will reference SQLitePCLRaw.bundle_e_sqlite3 version 2.0.6. This bundle doesn’t include the native SQLite reference, as demonstrated in the following image

This implies that you’ll need to rely on .NET 6’s native dependencies to include your custom version of lib.e_sqlite3, compiled specifically for WebAssembly. For more detailed information about native dependencies, please refer to the provided links.

https://github.com/egarim/XpoNet6WasmSqlite

https://learn.microsoft.com/en-us/aspnet/core/blazor/webassembly-native-dependencies?view=aspnetcore-6.0

If you’re using .NET 7 or later, your reference from Microsoft.Data.SQLite will depend on SQLitePCLRaw.bundle_e_sqlite3 version 2.1.5. This bundle provides several targets for the native SQLite library (e_sqlite3), as can see in the accompanying image.

This indicates that we can utilize SQLite on any platform supported by .NET, provided that we supply the native reference for SQLite.

Conclusion

 

Blazor WebAssembly and SQLite together offer a compelling option for developers looking to leverage the power of client-side data. Their combination enables the development of web applications with high performance, offline availability, and a unified language platform.

This potent mix empowers developers to rethink what’s possible with web application development, pushing the boundaries of what we can achieve with client-side data storage and manipulation. In a world where user experience is paramount, the coupling of these technologies truly helps in unleashing the full potential of client-side data.

Fake it until you make it: using custom HttpClientHandler to emulate a client server architecture

Fake it until you make it: using custom HttpClientHandler to emulate a client server architecture

by | Jul 17, 2023 | Blazor, C#, Data Synchronization, EfCore, WebAssembly

Last week, I decided to create a playground for the SyncFramework to demonstrate how synchronization works. The sync framework itself is not designed in a client-server architecture, but as a set of APIs that you can use to synchronize data.

Synchronization scenarios usually involve a client-server architecture, but when I created the SyncFramework, I decided that network communication was something outside the scope and not directly related to data synchronization. So, instead of embedding the client-server concept in the SyncFramework, I decided to create a set of extensions to handle these scenarios. If you want to take a look at the network extensions, you can see them here.

Now, let’s return to the playground. The main requirement for me, besides showing how the synchronization process works, was not having to maintain an infrastructure for it. You know, a Sync Server and a few databases that I would have to constantly delete. So, I decided to use Blazor WebAssembly and SQLite databases running in the browser. If you want to know more about how SQLite databases can run in the browser, take a look at this article.

Now, there’s still a problem. How do I run a server on the browser? I know it’s somehow possible, but I did not have the time to do the research. So, I decided to create my own HttpClientHandler.

How the HttpClientHandler works

HttpClientHandler offers a number of attributes and methods for controlling HTTP requests and responses. It serves as the fundamental mechanism for HttpClient’s ability to send and receive HTTP requests and responses.

The HttpClientHandler manages aspects like the maximum number of redirects, redirection policies, handling cookies, and automated decompression of HTTP traffic. It can be set up and supplied to HttpClient to regulate the HTTP requests made by HttpClient.

HttpClientHandler might be helpful in testing situations when it’s necessary to imitate or mock HTTP requests and responses. The SendAsync method of HttpMessageHandler, from which HttpClientHandler also descended, can be overridden in a new class to deliver any response you require for your test.

here is a basic example

public class TestHandler : HttpMessageHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // You can check the request details and return different responses based on that.
        // For simplicity, we're always returning the same response here.
        var responseMessage = new HttpResponseMessage(HttpStatusCode.OK)
        {
            Content = new StringContent("Test response.")
        };
        return await Task.FromResult(responseMessage);
    }
}

And here’s how you’d use this handler in a test:

[Test]
public async Task TestHttpClient()
{
    var handler = new TestHandler();
    var client = new HttpClient(handler);

    var response = await client.GetAsync("http://example.com");
    var responseContent = await response.Content.ReadAsStringAsync();

    Assert.AreEqual("Test response.", responseContent);
}

The TestHandler in this illustration consistently sends back an HTTP 200 response with the body “Test response.” In a real test, you might use SendAsync with more sophisticated logic to return several responses depending on the specifics of the request. By doing so, you may properly test your code’s handling of different answers without actually sending HTTP queries.

Going back to our main story

Now that we know we can catch the HTTP request and handle it locally, we can write an HttpClientHandler that takes the request from the client nodes and processes them locally. Now, we have all the pieces to make the playground work without a real server. You can take a look at the implementation of the custom handler for the playground here

Until next time, happy coding )))))