Crowdsourcing & Stack Overflow: the power of the Crowd

It's been 2 years since I last posted. Wow! Time flies and life changes...

In 2016 I started a Master's degree course at UFRJ - Federal University of Rio de Janeiro in the branch of Data and Knowledge Engineering (Databases) that is under the division of Computer and Systems Engineering department at COPPE\UFRJ.

Some background history: I tried to join this same master's program back in 2010. I posted about it here: Masters degree application essay UFRJ 2011.1. At that time I was not accepted in the program because the teacher responsible for the Software Engineering branch and that interviewed me, told me that I didn't have a research profile. I was more like an industry guy. I was upset of course when my application got rejected but I tried again in 2015 and guess what: I applied for the Software Engineering branch again and for the Databases branch too. I got approved in the tests for the two branches. The same teacher interviewed me for the Software Engineering after 5 years. She remembered me and said the same thing. She denied my application again. Thanks God the Database branch teachers were more receptive and accepted my enrollment. I love both areas and Databases involve everything related to data which attract my attention as well.

Lesson: don't give up on your dreams... if you want it with all your heart, go for it! It can take some time but it'll happen.

Back to the post title... as part of the grade, most of the 8 necessary disciplines have a final paper to be developed. This post is about the paper I did for CSCW - Computer Supported Cooperative Work or even Computer Aided Collaborative Work.

I hope it can shed some light regarding the cooperative work put together by the Stack Overflow developer community with the help of our mainstream computers and mobile devices.

Abstract—Crowdsourcing gained attention in the past few years
as a means to disseminate work to a crowd of people. People that
can be scattered all over the world. The Internet removed all the
barriers. Advances in Information Technologies and hardware
allowed the development of tools that aid in the division of the
work and tasks that need to be carried out. We are now in a stage
where what once was supposed to be inviable is proved to be viable
with remote work being done by disparate teams located in
different continents. Computer Supported Cooperative Work is
now a reality and this paper analyzes some of the concepts and use
of modern information systems technology to fill the gap between
a crowd of software developers and their day to day job questions
when applied to a questions and answers site namely Stack
Overflow. Each and every day millions of software developers
outsource their questions to knowledgeable peer developers. Stack
Overflow is the means that allows the knowledge transfer to
happen and it came in a moment when a huge amount of data was
being generated but it wasn’t well structured and organized for
future reuse. Stack Overflow was born and is now a great option
that offers a well-structured and exciting environment to ask and
get answers online.

The full paper in PDF format is available at:  https://drive.google.com/file/d/0B4SVxswDPXtwVzNFTWtSMkJoM3M/view?usp=sharing

Just got the C# gold badge at StackOverflow

This is a “marketing” post… Open-mouthed smile Just kidding.

This is just to inform that I got the C# #Csharp gold badge at #StackOverflow @StackExchange after 5 years and 6 months and 342 answers posted on the C# tag! What a milestone towards a so awesome programming language.

C# gold badge earned at StackOverflow

Once when I was starting back in 2003 a somewhat experienced coworker told me: this thing [ coding ] is not for you.

I proved him wrong!

The lesson to be learnt here is: never trust others when they say you are not capable even more when you are just starting something. Instead use that as a trampoline. Not easy but you must not give up. Keep pushing always.

People that motivate us are a rare kind. So use this post as a motivation and BEFORE that remember: put God first in everything you start in your life. I remember at that time in my life I talked to God and told him that I really liked this profession|area and asked for his help so that he could guide me and provide me wisdom. Here I’m today being able to write this post. How grateful I’m.

To top that, last week a cousin (13 years old) approached me and told me he’d like to follow the same path I followed and asked what I did to get where I’m today. I answered: you have to have a passion for what you do, strive for excellence and be prepared to spend some good amount of time trying and trying... as I wrote above: don’t give up. There’ll be for sure many rocks on the road… The Long and Winding Road as sung by The Beatles is a truth but if you persist in what you really believe then success is a matter of time.

Somehow sometime somewhere God will move things in your favor.

And we know that all things work together for good to them that love God, to them who are called according to his purpose. Romans 8:28 https://www.bible.com/bible/1/rom.8.28

A snapshot of my StackOverflow profile on the day I got the C# gold badge

http://stackoverflow.com/users/114029/leniel-macaferi

NPOI 2.0 - Support for right-to-left languages

This is the 5th post of a series of posts about NPOI 2.0.

This time we’re gonna see how easy it is to enable support for right to left languages in a spreadsheet.

Here’s a Microsoft page that describes Office’s right-to-left language features.

NPOI has got you covered too. The flag first appeared in NPOI 2.0 Alpha:

g. Support isRightToLeft and setRightToLeft on the common spreadsheet Sheet interface, as per existing HSSF support (poi-developers)

Let’s jump directly to the code with no further ado…

using NPOI.SS.UserModel;
using NPOI.XSSF.UserModel;
using System.IO;

namespace NPOI.Examples.XSSF.SetIsRightToLeftInXlsx
{
    class Program
    {
        static void Main(string[] args)
        {
            IWorkbook workbook = new XSSFWorkbook();

            ISheet sheet1 = workbook.CreateSheet("Sheet1");
            
            // Setting support for Right To Left
            sheet1.IsRightToLeft = true;

            sheet1.CreateRow(0).CreateCell(0).SetCellValue("This is a Sample");

            int x = 1;

            for(int i = 1; i <= 15; i++)
            {
                IRow row = sheet1.CreateRow(i);

                for(int j = 0; j < 15; j++)
                {
                    row.CreateCell(j).SetCellValue(x++);
                }
            }

            FileStream sw = File.Create("test.xlsx");

            workbook.Write(sw);

            sw.Close();
        }
    }
}

You can find the above code in the SetIsRightToLeftInXlsx sample project. I sent a pull request to Tony Qu so that this sample project gets added to NPOI’s GitHub repository.

Hope it helps!

Using MsDeploy publish profile .pubxml to create an empty folder structure on IIS and skip deleting it with MsDeploySkipRules

I’m going to share here a pretty nice deployment automation solution for when you need to have a defined set of folders in place when first deploying an app to IIS. On subsequent re-deployments the folder structure will be kept intact with any files users may have added to them.

Let’s work with a simple example: given an ASP.NET MVC app, we need a folder called Files and inside this folder there will be some pre-defined folders named: Folder 1, Folder 2 and Folder 3 with a child folder called Test. Something like this:

App root
|
---Files
   |
   ---Folder 1
   ---Folder 2
   ---Folder 3
      |
      ---Test

When deploying for the 1st time these folders are empty but the folder structure is mandatory let’s say because I’m using a file manager like elFinder.Net that expects that these folders exist on the server. Why? Because the ASP.NET MVC app has links pointing to these folders in some views. The folders should be ready to store files when the app is released. What also comes to my mind is the case where we need an existing Downloads/Uploads folder.

What’s more? We also want all this to happen while using Publish Web command from within Visual Studio and still keeping the option Remove additional files at destination checked:

Figure 1 - Visual Studio Publish Web with File Publish Options => Remove additional files at destinationFigure 1 - Visual Studio Publish Web with File Publish Options => Remove additional files at destination

This setting is nice because when you update jQuery NuGet package for example (jquery-2.1.1.js) it will send the new files to IIS server and will remove the old version (jquery-2.1.0.js) that exists there. This is really important so that the app keeps working as expected and don’t load the wrong version/duplicate files. If we don’t check that option we have to go to the server and delete the old files manually. What a cumbersome and error prone task!

What to do in this case where we want the deployment task do the work “automagically” for us with no human intervention? It’s seems like a lot of requirements and a task not so simple as “it appears to be”… Yep, it requires a little bit of MsDeploy codez.

Here’s what is working for me at the moment after finding some code pieces from here and there:

Given a  publish profile named Local.pubxml that sits here:
C:\Company\Company.ProjectName\Company.ProjectName.Web\Properties\
PublishProfiles\Local.pubxml

Let’s add the code blocks necessary to make all the requirements come to life:

<?xml version="1.0" encoding="utf-8"?>
<!--
This file is used by the publish/package process of your Web project. You can customize the behavior of this process
by editing this MSBuild file. In order to learn more about this please visit http://go.microsoft.com/fwlink/?LinkID=208121. 
-->
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
<AfterAddIisSettingAndFileContentsToSourceManifest>AddCustomSkipRules</AfterAddIisSettingAndFileContentsToSourceManifest>
    <WebPublishMethod>MSDeploy</WebPublishMethod>
    <LastUsedBuildConfiguration>Local</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <SiteUrlToLaunchAfterPublish />
    <ExcludeApp_Data>False</ExcludeApp_Data>
    <MSDeployServiceURL>localhost</MSDeployServiceURL>
    <DeployIisAppPath>SuperCoolAwesomeAppName</DeployIisAppPath>
    <RemoteSitePhysicalPath />
    <SkipExtraFilesOnServer>False</SkipExtraFilesOnServer>
<MSDeployPublishMethod>WMSVC</MSDeployPublishMethod> <AllowUntrustedCertificate>True</AllowUntrustedCertificate>
<EnableMSDeployBackup>False</EnableMSDeployBackup> <UserName /> <_SavePWD>False</_SavePWD> <LaunchSiteAfterPublish>True</LaunchSiteAfterPublish> </PropertyGroup> <PropertyGroup> <UseMsDeployExe>true</UseMsDeployExe> </PropertyGroup> <Target Name="CreateEmptyFolders"> <Message Text="Adding empty folders to Files" /> <MakeDir Directories="$(_MSDeployDirPath_FullPath)\Files\Folder 1" /> <MakeDir Directories="$(_MSDeployDirPath_FullPath)\Files\Folder 2" /> <MakeDir Directories="$(_MSDeployDirPath_FullPath)\Files\Folder 3\Test"/> </Target> <Target Name="AddCustomSkipRules" DependsOnTargets="CreateEmptyFolders"> <Message Text="Adding Custom Skip Rules" /> <ItemGroup>
      <MsDeploySkipRules Include="SkipFilesInFilesFolder">
        <SkipAction>Delete</SkipAction>
        <ObjectName>filePath</ObjectName>
        <AbsolutePath>$(_DestinationContentPath)\\Files\\.*</AbsolutePath>
        <Apply>Destination</Apply>
      </MsDeploySkipRules>

      <MsDeploySkipRules Include="SkipFoldersInFilesFolders">
        <SkipAction></SkipAction>
        <ObjectName>dirPath</ObjectName>
        <AbsolutePath>$(_DestinationContentPath)\\Files\\.*\\*</AbsolutePath>
        <Apply>Destination</Apply>
      </MsDeploySkipRules>

</ItemGroup>
</
Target> </Project>

This is self explanatory. Pay attention to the highlighted parts as they are the glue that make all the requirements happen during the publish action.

What is going on?

The property

<AfterAddIisSettingAndFileContentsToSourceManifest>
AddCustomSkipRules
</AfterAddIisSettingAndFileContentsToSourceManifest>

calls the target

<Target Name="AddCustomSkipRules"
DependsOnTargets="CreateEmptyFolders">
that in turn depends on the other task
<Target Name="CreateEmptyFolders">

CreateEmptyFolders take care of adding/creating the folder structure on the server if it doen’t exist yet.

AddCustomSkipRules contains two  <MsDeploySkipRules...>. One is to prevent deleting Files and the other prevents deleting the child folders.

Check the targets’ logic. They’re pretty easy to understand…

Note: make sure you don’t forget the

<UseMsDeployExe>true</UseMsDeployExe>

otherwise you may see this error during deployment:

Error    4    Web deployment task failed. (Unrecognized skip directive 'skipaction'. Must be one of the following: "objectName," "keyAttribute," "absolutePath," "xPath," "attributes.<name>.")        0    0    Company.ProjectName.Web

Simple as pie after we see it working. Isn’t it? Winking smile

Hope it helps!

As an interesting point, see the command executed by msdeploy.exe that gets written to the output window in Visual Studio:

C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe -source:manifest='C:\Company\Company.ProjectName\Company.ProjectName.Web\obj\
Local\Package\Company.ProjectName.Web.SourceManifest.xml' -dest:auto,IncludeAcls='False',AuthType='NTLM' -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -
skip:skipaction='Delete',objectname='filePath',absolutepath='\\Files\\.*' -skip:objectname='dirPath',absolutepath='\\Files\\.*\\*' -setParamFile:"C:\Company\Company.ProjectName\Company.ProjectName.Web\obj\Local\ Package\Company.ProjectName.Web.Parameters.xml" -retryAttempts=2 -userAgent="VS12.0:PublishDialog:WTE2.3.50425.0"

Checkout Git remote origin/branch (not master) to a local branch

Today I sent a small pull-request to the awesome Font-Awesome project at GitHub. To do that I had to learn some bits along the way that diverted from the common path when working with GitHub.

Font-Awesome has a separate branch called gh-pages which hosts the GitHub Pages (site) for Font-Awesome. The change I was about to do should go inside that branch and not on the common master branch.

Generally when cloning a forked repo from GitHub it will bring/checkout only the master branch.

Problem: how can I do work on that gh-pages branch?

This is the solution that worked just fine in my case…

Take a look at the following screenshot:

Figure 1 - Checkout remote branch (not master) named gh-pages to a local branch with upstream trackingFigure 1 - Checkout remote branch (not master) named gh-pages to a local branch with upstream tracking

Above I have Git bash (you can get it here)  opened in the cloned repo folder. I used GitHub for Windows to clone the repo I forked from Font-Awesome.

When you open Git bash it has the (master) branch already checked out as you see in “yellow” above.

Executing the command:

git remote show origin

we see that the repo has 3 Remote branches, namely:

4.1.0-wip
gh-pages
master

however only the master branch is configured for git pull and git push on my local computer. That’s ok if I would do some work on the master branch. If that was the case I already had everything correctly setup but remember that I need to do work on the gh-pages branch. To do so, the following command must be executed:

git checkout –b gh-pages –-track origin/gh-pages

What this command does is:

With git checkout we create a new local branch called gh-pages and immediately switch to it with the –b parameter. See that the branch name matches the one from the remote origin, in this case, gh-pages.

The –-track parameter means that this new local branch has its upstream set to a remote branch, in this case origin/gh-pages.

This is the result I wanted.

As a visual clue, in the following screenshot you see that the branch now shows in Visual Studio 2013 Team Explorer window as the selected one. This means it made it to my local computer. Now I can work on it and when done I can click on that Changes link and push my changes to the remote origin.

Figure 2 - gh-pages branch appears in Published Branches in Visual Studio 2013 Team Explorer windowFigure 2 - gh-pages branch appears in Published Branches in Visual Studio 2013 Team Explorer window

By the way, here’s the official doc reference for the checkout command. It depicts exactly what I wanted  to do but had no idea how to do it Don't tell anyone smile :

git checkout <branch>

To prepare for working on <branch>, switch to it by updating the index and the files in the working tree, and by pointing HEAD at the branch. Local modifications to the files in the working tree are kept, so that they can be committed to the <branch>.

If <branch> is not found but there does exist a tracking branch in exactly one remote (call it <remote>) with a matching name, treat as equivalent to

$ git checkout -b <branch> --track <remote>/<branch>

You could omit <branch>, in which case the command degenerates to "check out the current branch", which is a glorified no-op with a rather expensive side-effects to show only the tracking information, if exists, for the current branch.

Hope it helps!

Update

Ops… hehehe. Just know I read the guideline to contribute to Font-Awesome… too late!

I should have submitted the pull request against the 4.1.0-wip branch where wip means (work in progress).

OK… I went further and did all the process again (a brand new pull request) but this time targeting the correct branch. I even learned a new thing using Git Gui (installed with Git tools):

Figure 3 - Creating a tracking branch with Git Gui with the same name [ 4.1.0-wip ] as defined in originFigure 3 - Creating a tracking branch with Git Gui with the same name [ 4.1.0-wip ] as defined in origin

To get to Git Gui, right click the repo folder on the disk and select Git Gui in the context menu.

To create a new branch, click the Branch menu then Create…

See that I selected Match Tracking Branch Name and made Tracking Branch point to origin/4.1.0-wip. Then clicked the Create button.

What this does is the same thing achieved with that previous Git checkout command executed above. The only difference is that now it’s all visual.

It’s real nice to have options to get the job done.

Enjoy!

Backup Mac OS Skype chat history to the cloud

Today I wanted to save a backup copy of all my Mac OS Skype chat history.

First thing I did was look for where Skype stores chat history. This Skype’s support page provided the path:

How do I see my chat history in Skype for Mac OS X?

In my case the folder is here:

/Users/leniel/Library/Application Support/Skype/lenielmacaferi

Inside this folder there’s an SQLite database file called main.db (mine is 13.1 MB) with all chat history. For more details on that, check this answer on SuperUser:

Viewing the full Skype chat history

I wanted to backup that folder to the cloud more specifically to Google Drive. A new Google Search and then I found a great piece of software called MacDropAny by the young developer Sebastian Hallum Clarke! Basically it allows you to keep any folder synced to the cloud and supports many cloud service providers: Dropbox, Box.com, Copy, Google Drive, iClouDrive, MediaFire or Microsoft SkyDrive.

I got the MacDropAny 2.12.zip zip package and extracted the .app. Moved it to Mac OS Applications folder and opened it.

It asks you to choose a folder to sync with the cloud. I wanted to select that Skype chat history folder described above. Problem: that folder is stored inside the Library folder that is hidden by default in Mac OS and so it won’t show up in the list of folders available to MacDropAny. To fix that, follow what’s described here. Open Mac OS Terminal and type:

chflags nohidden ~/Library/

Now when asked to Choose a folder, Library will be available:

Figure 1 - Select Skype user folder (lenielmacaferi) to be backed up to the cloudFigure 1 - Select Skype user folder (lenielmacaferi) to be backed up to the cloud

After clicking Choose we need to select the Cloud service provider where to store the files:

MacDropAnyChooseCloudServiceProvider_thumb[1]Figure 2 - Selecting cloud service provider

Then it’s necessary to choose where to store the files in Google Drive. I created a new folder there called Skype and selected it:

Figure 3 - Choose an existing folder (Skype) on Google Drive or create New FolderFigure 3 - Choose an existing folder (Skype) on Google Drive or create New Folder

It then allows you to give a different name to the folder to be backed up. I kept the same name and clicked Sync.

Figure 4 - Name the folder copy (I kept the same)Figure 4 - Name the folder copy (I kept the same)

You may be prompted to enter your password to allow MacDropAny to access the Library folder. Provide the password if asked and you’re done.

Figure 5 - MacDropAny Folder Sync Success messageFigure 5 - MacDropAny Folder Sync Success message

This is the cloud’s power at its best.

Now I have a security copy of all my Skype history in a “safe” Thinking smile place.

If you would like, make a donation to Sebastian:

Figure 6 - Sebastian’s donation messageFigure 6 - Sebastian’s donation message

He did what’s not available as of now in Google Drive at least, that is, sync any given system folder to the cloud.

Big thank you Sebastian!

NPOI 2.0 - Converting Excel XLS documents to HTML format

This is the 4th post of a series of posts about NPOI 2.0.

This time we’re gonna see how easy it is to grab a .XLS file and then have it converted to HTML code.

Note: in NPOI 2.0.6 only XLS format is supported. XLSX support is coming in a future release (maybe 2.1) because it still needs more testing.

Enough said Winking smile, let’s get to the code:

using NPOI.HSSF.Converter;
using NPOI.HSSF.UserModel;
using System;
using System.IO;

namespace NPOI.Examples.ConvertExcelToHtml
{
    class Program
    {
        static void Main(string[] args)
        {
            HSSFWorkbook workbook;
            // Excel file to convert
            string fileName = "19599-1.xls";
            fileName = Path.Combine(Environment.CurrentDirectory, fileName);
            workbook = ExcelToHtmlUtils.LoadXls(fileName);


            ExcelToHtmlConverter excelToHtmlConverter = new ExcelToHtmlConverter();

            // Set output parameters
            excelToHtmlConverter.OutputColumnHeaders = false;
            excelToHtmlConverter.OutputHiddenColumns = true;
            excelToHtmlConverter.OutputHiddenRows = true;
            excelToHtmlConverter.OutputLeadingSpacesAsNonBreaking = false;
            excelToHtmlConverter.OutputRowNumbers = true;
            excelToHtmlConverter.UseDivsToSpan = true;

            // Process the Excel file
            excelToHtmlConverter.ProcessWorkbook(workbook);

            // Output the HTML file
            excelToHtmlConverter.Document.Save(Path.ChangeExtension(fileName, "html"));
        }
    }
}

The code above was taken from ConvertExcelToHtml sample project.

Here’s the sample spreadsheet open in Excel for Mac that was used as input:

Figure 1 - Excel spreadsheet to be converted to HTML code
Figure 1 - Excel spreadsheet to be converted to HTML code

This is the HTML generated:

Figure 2 - HTML output code generated from the conversion
Figure 2 - HTML output code generated from the conversion

Note that I used a screenshot above but it depicts the .html file (check the browser’s address bar).

Enjoy and have fun!