Taras codes

MSBuild configuration tailored for every member of a team

hello@taras.codes — Sat, 17 Aug 2019 12:17:10 GMT

Sometimes we need to include extra steps in the build process on a single machine involved in a project. Often this is based not on an environment specification itself, but rather on developers' personal preferences on how to approach their work. Small teams working on small projects have the highest probability of facing such problems since enforcing strict development standards for them might be an overkill.

The $(ComputerName) environmental variable in a csproj file comes to the rescue here:

<Target Name="PreBuild" AfterTargets="PreBuildEvent" Condition="Exists('$(ComputerName).prebuild.bat')">
    <Exec Command="$(ComputerName).prebuild.bat" />
</Target>

Include this target in your project's csproj file to run any script from YOUR-PC-NAME.prebuild.bat file in the root folder of the project.

Notice a Condition on the target: a script is to be called only in case a file with your machine's name exists in the root folder. That means that any team member can opt-out of this step without any additional settings on their side.

You can also put *.prebuild.bat line in your .gitignore file, because nobody likes your personal stuff in the source code you share with a team.

Something that you may face here is a blocked MSBuild if you run a task that takes significant time to finish while not being required for the rest of the build process or a task that does not exit at all without user interaction (e.x., your ng serve or webpack --watch process).

This StackOverflow answer suggests a very useful ExecAsync implementation:

<!--Launch a Process in Parallel-->
<UsingTask TaskName="ExecAsync" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
    <ParameterGroup>
        <!--The file path is the full path to the executable file to run-->
        <FilePath ParameterType="System.String" Required="true" />
        <!--The arguments should contain all the command line arguments that need to be sent to the application-->
        <Arguments ParameterType="System.String" Required="true" />
    </ParameterGroup>
    <Task>
        <Code Type="Fragment" Language="cs">
            <![CDATA[
                System.Diagnostics.ProcessStartInfo processStartInfo = new System.Diagnostics.ProcessStartInfo(FilePath, Arguments);
                processStartInfo.UseShellExecute = false;
                System.Diagnostics.Process.Start(processStartInfo);
            ]]>
        </Code>
    </Task>
</UsingTask>

Now you can run your script with the following target:

<Target Name="PreBuild" AfterTargets="PreBuildEvent" Condition="Exists('$(ComputerName).prebuild.bat')">
    <ExecAsync FilePath="cmd.exe" Arguments="/c $(ComputerName).prebuild.bat" />
</Target>

which will give you the same result without blocking the rest of the build process.

Bonus: what's going on in TARAS-LAPTOP.prebuild.bat

tasklist /nh /fi "imagename eq code.exe" | find /i "code.exe" > nul || code client

When I start my Debug build, an instance of VS Code is launched in the ./client folder if there are none running already. Meanwhile, VS Code is configured to build Typescript files and watch for changes in them via yukidoi.blade-runner extension.

Bonus 2: just executing a Target that does not involve any scripts on a specific machine can be done with a file used as a flag:

<Target Name="CustomAction" AfterTargets="AnotherAction" Condition="Exists('$(ComputerName).msbuild.flag')">
    ...
</Target>

Now this Target is executed only if a file with a name YOUR-PC-NAME.msbuild.flag is present.

Get LinkedIn organization follower count

hello@taras.codes — Mon, 08 Jul 2019 14:23:04 GMT

For some reason, all social media APIs tend to be the least engineer-friendly* _{(*personal opinion)}and are a real pain to work with. Regular international scandals around the misuse of users' personal data do not make things easier and usually lead to even more restrictions on data access for developers. Unless you use a member authentication token (aka social media sign-in button) in your app, most platforms won't grant you access to pages' public data or will at least make it difficult by requiring you to fill in enormous forms and provide demos to explain your use cases.

Security of your personal data on the web is good, don't get me wrong. But what if you need some basic public info to develop an app that will bring value to users and at the same time there doesn't seem to be a logical way to include a "Sign-in with your social media account" process?

LinkedIn page follower count

A common use case of providing a page follower count by a platform to anonymous users is a "Follow us" button that you can put on your website. LinkedIn has one as well. It looks like this:

That's a follower count for Microsoft if you are wondering. Here's a code to create one:

<script src="https://platform.linkedin.com/in.js" type="text/javascript"> lang: en_US</script>
<script type="IN/FollowCompany" data-id="1035" data-counter="bottom"></script>

where 1035 is the ID for Microsoft's LinkedIn page.

Let's put it on a web page and see what happens on the Network tab of DevTools.

Here it is - a public URL to retrieve a page's follower count without OAuth and/or request limitations:

https://www.linkedin.com/pages-extensions/FollowCompany?id={companyId}&counter=bottom

This is something we can work with. Let's write some code.

"""
linkedin_followers.py
"""
from sys import argv
from pyquery import PyQuery
import re
pq = PyQuery(url=f'https://www.linkedin.com/pages-extensions/FollowCompany?id={argv[1]}&counter=bottom')
widget_text = pq.text() # equivalent of <document.body.innerText> in JS
follower_count = int(re.sub(r'\D', '', widget_text)) # remove everything except digits and cast to int
print({ 'raw': widget_text, 'count': follower_count })

This will give us the following result:

$ python linkedin_followers.py 1035
{'raw': 'Follow\n7,750,599', 'count': 7750599}

LinkedIn company ID

What if you want a number for microsoft, and not some 1035 page? Well, I don't have a pretty solution yet. Here are some things you can try depending on your use case.

Prompt to provide a LinkedIn ID as input

If you are able to provide input as LinkedIn page IDs, you will have a solution ready. In case it is a user who will fill in the input field, you could add an instruction on getting an ID near your input field. The official LinkedIn documentation itself provides users with the following tip to set up a "Follow" button:

As a company page administrator, your Company ID can be retrieved by navigating to the admin section of your company page. For example, the LinkedIn Company Admin Page is https://www.linkedin.com/company/1337/admin/. We will use the Company ID 1337 in our example.

Crawl LinkedIn to get an ID

This method should not be your preferred solution. You should also be aware that sometimes LinkedIn will just call you out on being a bot and ask for a CAPTCHA. Sometimes though it will not. So you can automate this to some degree.

First thing you should consider is that company ID won't change over time, so once obtained should be stored in a database with a corresponding company name instead of being requested each time.

First thing you will need to crawl LinkedIn is a li_at cookie to identify yourself as a user. To get one, you need to emulate login. I find Nightmare (a Node.js library with Electron under the hood) to be very useful in such tasks. Here is an example of how you can use it for the LinkedIn sign-in process:

/**
 * linkedin_login.js
 */
var Nightmare = require('nightmare');

var TIMEOUT = 5000; //ms
var SHOW_BROWSER = true;

var linkedinLogin = function(email, password) {
    var nightmare = new Nightmare({
        waitTimeout: TIMEOUT,
        show: SHOW_BROWSER
    });
    return nightmare
        .goto('https://www.linkedin.com/uas/login')
        .wait('#username') // wait until sign-in form appears
        .insert('#username', false) // clear
        .insert('#username', email)
        .insert('#password', false) // clear
        .insert('#password', password)
        .click('.btn__primary--large')
        .wait('.content')
        .cookies.get('li_at')
        .end();
};

var argvCount = process.argv.length;
var email = process.argv[argvCount - 2];
// clearly, you should not input your password as a command-line argument in a real world out there
var password = process.argv[argvCount - 1];
linkedinLogin(email, password).then(console.log);

Let's launch the script:

$ npm install nightmare
$ node linkedin_login.js [email] [password]

If you have specified SHOW_BROWSER = true (which is optional, but useful for debugging purposes), you will see an Electron window for a brief moment:

and your result after that:

{ name: 'li_at',
  value:
   'COOKIE_VALUE',
  domain: '.www.linkedin.com',
  hostOnly: false,
  path: '/',
  secure: true,
  httpOnly: true,
  session: false,
  expirationDate: 1594297571.568987 }

Like with the company ID, if you are going to pull this value before each new request, the LinkedIn server's patience might run out quickly, so you better store it somewhere before its expirationDate.

Now for the company ID itself. It seems to be exposed through various links on the page. One of them is this link:

So why don't we pull the ID right from it? Here's another Nightmare library approach:

/**
 * linkedin_company.js
 */
var Nightmare = require('nightmare');

var TIMEOUT = 5000; //ms
var SHOW_BROWSER = true;

var getLinkedinCompanyId = function(companyName, authCookieValue) {
    var nightmare = new Nightmare({
        waitTimeout: TIMEOUT,
        show: SHOW_BROWSER
    });
    return nightmare
        .goto('https://www.linkedin.com')
        .cookies.set('li_at', authCookieValue)
        .goto('https://www.linkedin.com/company/' + companyName)
        .wait(function () {
            return document.getElementsByTagName('a');
        })
        .evaluate(function() {
            var links = Array.from(document.getElementsByTagName('a'))
                .filter(function (e) { return e.href.indexOf('facetCurrentCompany') >= 0; });
            return links[links.length - 1].href
        })
        .end()
        .then(function (employeesUrl) {
            var parsedUrl = new URL(employeesUrl);
            var currentCompanyParam = parsedUrl.searchParams.get('facetCurrentCompany');
            return parseInt(JSON.parse(currentCompanyParam)[0]);
        });
};

var argvCount = process.argv.length;
var company = process.argv[argvCount - 2];
var cookie = process.argv[argvCount - 1];
getLinkedinCompanyId(company, cookie).then(console.log);

Now let's run it with the cookie value we stored:

$ node linkedin_company.js microsoft "COOKIE_VALUE"
1035

And here we have it: 1035 is our ID for the microsoft page. We can do it again with any page, no matter the relation between our logged-in user and a company.

$ node linkedin_company.js apple "COOKIE_VALUE"
162479

Now we can use it with our linkedin_followers.py script:

$ python linkedin_followers.py 162479
{'raw': 'Follow\n6,983,914', 'count': 6983914}

Hosting an ASP.NET Core application on Raspberry Pi 3

hello@taras.codes — Wed, 15 May 2019 15:11:19 GMT

.NET Core is gradually becoming quite a popular platform these days, and one of the reasons for that is its cross-platform features and a neat command line interface which allows creating boilerplates, running and building applications with simple yet highly configurable commands.

So one time, after spending a weekend looking around what ASP.NET Core offers, I decided to create a sandbox for my .NET Core experiments using a Raspberry Pi I happen to have on my hands. Naturally, I expected setting up to be something like: install the runtime, dotnet app.dll. Well... it was not the case. But after spending several evenings trying different configurations from the web (and ending up with tons of ridiculously undescriptive error messages), I came up with the following steps that worked just fine.

My setup

Raspberry Pi 3 Model B v1.2
Ubuntu Server 18.04.2 installed on Raspberry Pi with enabled openssh-server
My laptop with preinstalled .NET Core SDK

Creating a boilerplate app

Once you have .NET Core SDK installed, you can list available templates with:

dotnet new -l

For this example, let's create a webapp. Since I have multiple SDK versions installed, I'm going to specify that I want a .NET Core 2.2 application using a global.json file:

mkdir test
cd test
dotnet new globaljson --sdk-version 2.2.107

Then to generate the app itself:

dotnet new webapp

You can make sure the application has been created properly right away by running:

dotnet run

Now your boilerplate app is running on http://localhost:5000

Hosting on Pi. The easy way

The easiest way to run your app on a Raspberry Pi is using a .NET Core's self-contained deployments (SCD). This way an appropriate runtime will be shipped with your application - ready to run with a single console command.

Let's create one of those:

dotnet publish -f netcoreapp2.2 -c Release -r ubuntu.18.04-arm64 -o ~/test_build ./

The -r parameter is the key here. You can use it to specify a runtime identifier and state that you need a self-contained deployment. A full list of available runtime identifiers can be found here. I could go with a linux-arm64 runtime here, but I'm specifying an ubuntu.18.04-arm64 runtime because I can.

Expect the build to be quite large since it contains everything you need to run your application on a specified system.

Now I'm going to copy the published app to my Raspberry over ssh:

scp -r ~/test_build ubuntu@<raspberry host>:/home/ubuntu/testapp

And I'm ready to launch the server on my Raspberry:

cd /home/ubuntu/testapp
chmod +x test
./test

At this point, you'll see your ASP.NET Core server running on http://localhost:5000. Pretty cool. Now let's make it a systemd service.

We'll need to create a /etc/systemd/system/testapp.service file with the following contents:

[Unit]
Description=An example ASP.NET Core app
 
[Service]
WorkingDirectory=/home/ubuntu/testapp
ExecStart=/home/ubuntu/testapp/test
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=testapp
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
 
[Install]
WantedBy=multi-user.target

Then run:

sudo systemctl enable testapp.service
sudo systemctl start testapp.service

Now we can try navigating to our homepage...

curl http://localhost:5000

...and also watch the server's output:

journalctl -u testapp -f

Ok, but how about hosting this on port 80? We can do an nginx reverse proxy here:

sudo apt install nginx
sudo service nginx start

Then replace contents of the /etc/nginx/sites-available/default with:

server {
    listen        80;
    server_name   <your host name / public ip>;
    location / {
        proxy_pass         http://localhost:5000;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection keep-alive;
        proxy_set_header   Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

After that let's check the syntax of the file and apply changes with:

sudo nginx -t
sudo nginx -s reload

Now for our project itself. We'll need to edit Startup.cs file and append the following code to the Startup.Configure method:

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
});

before the application's authentication routine is set up (ex. before app.UseAuthentication() call). Save and redeploy as the first time. Restart the server:

sudo systemctl restart testapp.service

And you have your application running on port 80 of your Raspberry Pi 3.

Hosting on Pi. Shared runtime

Now, it is not very comfortable to generate SCDs for each update and each application you need to run on the Raspberry: they are large, they take longer to deploy and they can only be used for a single system configuration, specified with a runtime identifier. So can we have a shared runtime installed and use smaller cross-platform framework-dependent deployments (FDD)? Kind of.

On the extended list of available .NET Core downloads, you can find a .NET Core runtime (or even SDK) binaries for a Raspberry Pi 3. You're looking for Linux .NET Core Binaries for ARM64. Notice how there's no link for ASP.NET framework for ARM64? Yeah. The good news, we'll have one with .NET Core 3.0 where full native support of ARM64 is promised. With .NET Core 2.2 you are limited to ensuring your ASP.NET application deployment carries everything it needs or using ARM32 ASP.NET Core binaries by enabling your Raspberry to run them.

If you still want to have at least .NET Core runtime on your Pi, here's a way to install that:

curl -SL -o dotnet-runtime.tar.gz https://download.visualstudio.microsoft.com/download/pr/7aca89ca-5196-4b89-93bc-1ee1eeb251d7/ca4ff94c8692a6846a756fc07174974d/dotnet-runtime-2.2.5-linux-arm64.tar.gz
sudo mkdir /usr/share/dotnet
sudo tar -zxf dotnet-runtime.tar.gz -C /usr/share/dotnet
rm dotnet-runtime.tar.gz
sudo ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet

You are also free to select another binaries version from the download page. To install the SDK, make sure you also have these requirements:

sudo apt install liblttng-ust0 libcurl4 libssl1.0.0 libkrb5-3 zlib1g libicu60

I have also found a very good tip about setting up a .NET Core environment in this note.

Pro tip: Check out .NET Core Docker files to determine the exact instructions for installing .NET Core builds, for example .NET Core 2.2 ARM32 SDK Dockerfile.

Once your binaries are set up, you can check your global dotnet availability by running

dontent --info

How self-contained SCDs are?

Somewhat self-contained. You are ready to go with a single command in case of a simple ASP.NET Core MVC app (like the one created from webapp template), but do not expect dotnet to resolve every dependency for you by stating a target platform in a publish script.

For example, while trying to run a Miniblog.Core (a nice little stand-alone blog engine for .NET Core) SCD instance, I found out that it expects a globally installed libsass to build its scss files with the WebOptimizer middleware.

sudo apt install libsass-dev

did the trick here.