by Joche Ojeda | Oct 10, 2021 | Data Synchronization
In the last post, we talked about what are deltas and how by using them we can synchronize data structures.
So, in this post, I will describe the necessary parts needed to implement Delta-based synchronization, let’s start
- Data Object: any database, object, graph, or file system that we are tracking for synchronization
- Delta: a delta is a data difference between the original and the current state of a data object
- Node: is a point of the synchronization network, there are 2 types of nodes
- Client node: a node that is able to produce and process deltas
- Server node: a node that is used only to exchange deltas, it can optionally process deltas too.
- Delta Store: storage where you can save deltas so you can later exchange them with other nodes
- Delta Processor: a utility that helps you includes the deltas created in other nodes in your current copy of the data object
Now let’s implement this concept to synchronize a database
Ok so each database that we want to synchronize will be a client node and a client node should be able to produce and store deltas and to process deltas produced by other nodes, so our database node should look like the following diagram
The server node can be as simple as just delta storage exposed in an HTTP API as you can see in the basic server node diagram or use several delta storages and delta processors as show on the complex server node diagram
And with those diagrams, I finish this post, in the next post we will implement those concepts in C# code
by Joche Ojeda | Oct 10, 2021 | Data Synchronization
To Synchronize data is one of the most challenging tasks out there, especially if you are working with LOB applications
There are many ways to synchronize data, the most common technique is to compare records by modification date and then merge the data to create a master record.
Here the main problem is that you have to have a way to compare each type of record, so it gets cumbersome as soon as the number of types in your application begins to grow.
Also, you have to have a log of what gets created and what gets deleted so you can do the same changes on the other nodes
Delta-based synchronization
delta-based synchronization is a problem of identifying “what changed” between each execution of a sync process
A directed delta also called a change, is a sequence of (elementary) change operations which, when applied to one version V1, yields another version V2 (note the correspondence to transaction logs in databases). In computer implementations
In delta synchronization, there are 2 main tasks
- Record the deltas (or small difference of data between nodes)
- Transmit these differences to the other nodes so they can process them
Implementing Delta-based synchronization for relational databases
The schema above represents a blog’s database, it’s a really simple schema so its easy to understand, now this is the scenario
We have the main database that we will call “Master” and 2 other databases named client A and client B.
Now let insert data in the master
Each DML statement should be converted in a delta (or a data difference)
Δ1
Δ2
Δ3
Copy deltas Δ 1, Δ 2, Δ 3 to the clients
So, after processing the deltas on each client, the data in all databases should look like the picture below
So that’s it for this post, in the next post we will be examing each part that is necessary to do delta-based synchronization
by Joche Ojeda | Oct 29, 2020 | Linux, netcore, Ubuntu, XAF
If you are running Xaf Blazor in ubuntu 18.04 you might have seen the following exception
The type initializer for ‘Gdip’ threw an exception.
at DevExpress.ExpressApp.Actions.ActionBase.OnHandleException(Exception e) at DevExpress.ExpressApp.Actions.ActionBase.ExecuteCore(Delegate handler, ActionBaseEventArgs eventArgs) at DevExpress.ExpressApp.Actions.PopupWindowShowAction.DoExecute(Window window) at DevExpress.ExpressApp.Actions.PopupWindowShowAction.DialogController_Accepting(Object sender, DialogControllerAcceptingEventArgs e) at DevExpress.ExpressApp.SystemModule.DialogController.Accept(SimpleActionExecuteEventArgs args) at DevExpress.ExpressApp.SystemModule.DialogController.acceptAction_OnExecute(Object sender, SimpleActionExecuteEventArgs e) at DevExpress.ExpressApp.Actions.SimpleAction.RaiseExecute(ActionBaseEventArgs eventArgs) at DevExpress.ExpressApp.Actions.ActionBase.ExecuteCore(Delegate handler, ActionBaseEventArgs eventArgs)
The error is caused by missing dependency, so the DotNet runtime itself will throw that exception. Also, I want to highlight that the exception is not related to XAF, you can read more about this problem here https://github.com/dotnet/runtime/issues/27200
To get the missing dependency just open a console and run the following commands
sudo apt-get update -y
sudo apt-get install -y libgdiplus
by Joche Ojeda | Sep 14, 2020 | Linux, netcore, Ubuntu
Here are some recommendations to host your new shiny aspnet core app on Linux in this case in Ubuntu 18.04
First, create a user with the name aspnetapp
sudo adduser myaspnetapp
after executing the command, you will have a new folder in your home directory the folder will have the same name as your username so in this case “myaspnetapp”
now let’s SSH to with the new user you just created you can do that using your favorite SSH client, for example, if you are using windows you can use putty
when you log in with the new user you will be in its home folder, now we can create a folder called app with the following command
mkdir app
your folder structure should look like this now
/home/myaspnetapp/app
Now we are ready to upload the files. By now should have already compiled and publish your application to run in Linux, if you have not done that yet then you should take a look to this article https://www.jocheojeda.com/2019/06/10/how-to-create-a-self-contained-netcore-3-console-application/
There are many options to upload a zip file but I think is the best way is to use the secure copy command from linux “scp”, I won’t explain how you should call the scp command but if you are using windows you can run that command from the WSL console and if you are using Linux the command is already built-in, anyway here is an article about it https://linuxize.com/post/how-to-use-scp-command-to-securely-transfer-files/
Here I will write an example of how the scp command should look like and you adjust it to your needs
scp publish.zip myaspnetapp@200.35.15.25:/home/myaspnetapp/app
so that command above will do the following, it will copy the file publish.zip from the local folder to a server running on the following the IP 200.35.15.25 and the folder “/home/myaspnetapp/app”
now let’s unzip the content of the folder zip with the following command
unzip publish.zip
What we have done so far:
- We have created a user in the OS
- We have created a folder to host our application within the user home folder
- We have uploaded a zip file containing our application the folder “/home/myaspnetapp/app”
Now that the app is already in the server we need to change the permission of the folder where the app lives to 0777, you can learn more about Linux file system permissions here https://www.guru99.com/file-permissions.html
Creating a service to monitor the app
The next step to monitor our app is to use systemd is an init system that provides many powerful features for starting, stopping, and managing processes.
Let’s start by creating a service file in the following path “/etc/systemd/system/”
You can do that with the following command:
sudo nano /etc/systemd/system/MyExecutableFile.service
here is how the content of the file should look like
[Unit]
Description=A description of your app
[Service]
WorkingDirectory=/home/myaspnetapp /app
ExecStart= /home/ myaspnetapp /app/MyExecutableFile
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier= MyExecutableFile
User=apache
Environment=ASPNETCORE_ENVIRONMENT=Production
[Install]
WantedBy=multi-user.target
Here is a little explanation of what you might need to change if the file above
WorkingDirectory: this is the working directory, usually is the same where the app lives
ExecStart: This is the executable file how what will you write here will depend on your application if its self-contained you just need to write the name the full path of the executable file otherwise you need to write the path to the dotnet runtime and then the path to your dll as show below:
/usr/local/bin/dotnet /var/www/helloapp/helloapp.dll
RestartSec: this is the time to wait before trying to restart the app after if the process crashes
SyslogIdentifier: the app identifier for sys logs
User: this is really important since the app will run under this user privileges, so you need to make sure that the user exists and that is able to access the files needed to start the app
That is all that we need for the service file now we need to go back to the console and enable our new service, you can do that with the following command
sudo systemctl enable MyExecutableFile.service
To start and stop the service you can use the following commands
//To Start
sudo systemctl start MyExecutableFile.service
//To Stop
sudo systemctl status MyExecutableFile.service
by Joche Ojeda | Sep 11, 2020 | Linux, Ubuntu
Lately, I have been playing a lot with TestCafe which is a testing tool for website, there are 2 parts of TestCafe, the first part is TestCafe studio which is a tests editor tool where you can create or edit new tests that will be eventually used in the TestCafe test runner, both TestCafe and the test runner runs on Windows, MacOS, and Linux.
So what we are going to do today is to install TestCafe on Ubuntu Linux 18.04, in this tutorial we will only install the test runner because, in the end, my main goal is to use TestCafe to emulate the load on a server, so let’s get started
First, we need to install NodeJS and NPM, this is kind of tricky depending on your OS version, for example, if you run this command :
sudo apt -y install nodejs
in Ubunto 18.04 you will end up installing NodeJS version 8 something… that’s too old for what we need to do, so first let’s install the latest version of NodeJS, in this case, this is version 12
Installing NodeJS and NPM
1) First, let’s update our repository information
sudo apt update
sudo apt -y upgrade
2) Manually add Node.js APT Repository
sudo apt update
sudo apt -y install curl dirmngr apt-transport-https lsb-release ca-certificates
curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
3) now let’s install NodeJS
sudo apt -y install nodejs
4) This step is optional, if you want to verify the versions of NodeJS and NPM you can use the following commands:
node --version
npm --version
Installing TestCafe
To install TestCafe you only need to execute one command, so here it is
npm install -g testcafe
That was easy !!!
That is everything for this post, see you!!!
