When developing a chess game today, I stumbled upon a peculiar behavior in Chromium browsers while implementing draggable elements. If you’ve ever noticed that dragging an element causes it to inherit its parent’s background, you’re not alone. This odd effect can be quite a nuisance, but fortunately, there are ways to fix it.

Understanding the Issue

The issue arises when a draggable element seems to take on the background of its parent element during the drag action. This can lead to unexpected and unwanted visual results, especially if the parent element’s background is distinct or patterned.

To understand why this happens, let’s delve into some technical insights:

• The HTML draggable attribute (draggable=”true”) seems to force the element to inherit the parent’s background.

• According to the HTML Living Standard, the drag data store default feedback is dependent on the user agent (browser). This means different browsers might handle draggable elements differently.

  Here’s a snippet from the HTML Living Standard that highlights this:

“Update the drag data store default feedback as appropriate for the user agent. If the user is dragging the selection, then the selection would likely be the basis for this feedback; if the user is dragging an element, then that element’s rendering would be used; if the drag began outside the user agent, then the platform conventions for determining the drag feedback should be used.”

Because of this browser-dependent behavior, the default feedback during a drag action can vary, making it challenging to create a consistent user experience.

Fixing the Issue

Through some research and experimentation, I found two effective ways to fix this issue:

• Using position: relative and z-index: By setting the draggable element’s position to relative and applying a z-index, you can ensure it retains its own background.

[draggable] { position: relative; z-index: 1; }

• Using CSS Transforms: Applying a small transformation to the draggable element can also resolve the issue.

[draggable] { transform: translate(0,0); }

Why These Fixes Work

1. position and z-index: By setting the position to relative and giving it a z-index, you force the element to create a new stacking context. This prevents the draggable element from inheriting the parent’s background during the drag operation.

2. CSS Transforms: Using a small transformation disrupts the default rendering process enough to ensure that the draggable element maintains its own background. The translateX and translateY values can be minimal and should not visibly affect the element’s position.

Conclusion

Browser inconsistencies can be frustrating, especially when dealing with visual feedback during drag-and-drop operations. By understanding the underlying causes and applying these CSS fixes, you can ensure your draggable elements display correctly across different browsers.

Have you encountered any other weird browser behaviors? Share your experiences and solutions in the comments below!

This post is also available on my portfolio blog, so be sure to visit there for more updates and insights.

Photo by Denny Müller on Unsplash

The post Fixing the Draggable Element Background Issue in Chromium Browsers appeared first on ProdSens.live.

Experimenting with your lead generation content is crucial to build a strong content strategy. Fresh types of content can expand your reach to attract more — and possibly better — leads.

The best way to shake up your lead generation content strategy is to take a deeper look into the top types of content that will help you collect new leads.

They’re ranked by effectiveness for 2023 based on statistics from our most recent survey of over 1,200 marketing professionals.

Because those statistics point so clearly to the growing shift toward social media marketing, we’ll also share data and tips on leveraging the top four social media channels for lead generation.

Top Lead Generation Content Types

The best lead generation content types can help your business’s reach grow by leaps and bounds. Here are the types of content you should consider using.

Short-Form Videos

Video tops the list in lead-generating effectiveness as overall marketing trends continue to move toward social media platforms.

Short-form videos are soaring in popularity across all social channels, including Facebook Reels, Instagram Reels, YouTube Shorts, and TikTok.

Short-form video has the highest ROI of any of the marketing trends, and 90% of marketers already using it planned to maintain or increase their investment in 2023.

Image Source

In terms of creating video content, it’s up to you to decide whether you want to produce, shoot, and edit the video in-house or hire a professional.

Short-form video taken with a smartphone camera is on-trend, as the content appears personable and easy to connect with. It’s also easier to create content on the go to capture interesting events.

To learn what kind of video content your target audience likes, you’ll need data indicating which social media platform(s) they use most. Use that platform’s best practices and follow reputable guides on video content creation to get started.

Influencer Marketing

Let’s take a moment to talk about social media marketing trends and how Gen Zers and Millennials fit into the picture.

In the last three months, 33% of people within these groups have bought a product due to an influencer’s recommendation.

It’s no wonder that influencer marketing, which is prevalent across social media platforms, has the second highest ROI in terms of lead generation, just after short-form videos.

Influencers are generally considered to be experienced peers who are knowledgeable, trustworthy, and entertaining.

Marketers who can leverage an influencer’s followers with integrity can create compelling — and high-selling — content for a pre-built audience.

Within Instagram, the highest ROI comes through integrated shopping tools such as Instagram Shop where users can make purchases without even leaving the social media app.

What’s more, in the past 3 months, 22% of the Gen Z demographic has skipped websites entirely for customer service, opting to reach out to brands via direct messaging (DMs) on social media. If your brand isn’t already getting social, you’re missing out.

Website/Blog/SEO

Blogging not only drives more traffic to your website but can also become a major source for lead generation down the road.

In 2022, 88% of marketers surveyed said they plan to maintain or increase their investment in blog/SEO the following year. It ranks third in ROI, remaining tried and true.

Blogging is effective, accessible, and for some of us it’s even enjoyable. But how do you convert blog readers into leads?

The most common way to turn blog visitors into dedicated subscribers is by simply asking for their email addresses in order to send them notifications when you post new content. It’s more convenient to have content delivered to your inbox than to have to seek it out.

Create a clear and central call to action inviting readers to subscribe. Add a one-step form to your blog pages, like the American Writers Museum does with its blog:

Image Source

Another great option is to incentivize the blog subscription for new subscribers with free offers. That could be anything from a one-time welcome discount, a free gift, access to useful tools for those in your industry, or more content such as gated ebooks.

Blogging for leads is a big topic to explore, but I suggest this targeted how-to guide on converting visitors into subscribers if you’d like to learn more on that specific topic.

Email Marketing

Very much still in the game, lead generation via email marketing ranks 4th in marketer use by a minuscule 1% behind website/blog/SEO.

The hot topic in email marketing for lead generation is using AI and process automation connected to CMS data to make your email blasts personalized. This also saves a lot of time, and personalization improves metrics. Here are 23 personalized email examples I love.

Interestingly, when those surveyed think forward to the possibility of a recession this year, email marketing is one of the top channels that marketers expect to see budget cuts for.

In comparison, Blog/SEO is first in line for keeping its budget in the case of an economic downturn, followed by current marketing darlings podcast, direct mail, and organic social.

Additional Data and Expert Tips

Top Trending Lead Gen Social Media Channels

Image Source

It’s clear from the data that marketing will continue to trend toward social media at a faster rate than in previous years.

Below is a current snapshot of the top four trendy social media channels for your lead generation content strategy includes (in order of survey ranking).

Facebook

The platform leads in ROI and was predicted to see the most investment in 2023, as 1 in 4 marketers plan to invest for the first time this year.

Our combined tip for Facebook and Instagram is to explore and leverage the Meta Ads Manager that works across both of these platforms as well as FB Messenger to save time via integration and data sharing.

Instagram

In 2022, 58% of marketers surveyed already leveraged IG, and the platform was expected to see high first-time use in 2023.

YouTube

The 2022 survey revealed that this platform will see the most overall growth in 2023, with 91% of the marketers who already use it planning to maintain or increase their current investment.

29% of marketers who didn’t use it planned to try it for the first time in 2023. There are at least seven places to optimize text for your video, so make a list of long and short-tail keywords related to your content and fold them in everywhere you can.

TikTok

56% of marketers surveyed who already leverage TikTok planned to increase their investment in 2023, 16% plan to invest more this year than they ever have before, and 28% of non-users plan to start.

This represents the highest growth in investment among all social media channels. There are two lead generation forms available through TikTok — one is native to the app, or you can connect your website’s form if it is compatible.

By experimenting with different types of lead generating content, you can observe which ones resonate with your audience and convert the most leads. You can always do more of what works, but never get too comfortable!

Keep Experimenting with Lead Generation Content Strategy

8 in 10 marketers shared that they have seen more change in their industry in the last three years than in the previous 50 combined.

Data-driven marketers who are prepared to pivot and adapt will find the most success with their lead generation content strategies this year and beyond.

Editor’s Note: This post was originally published in October 2014 and has been updated for freshness, accuracy, and comprehensiveness.

The post Lead Generation Content: Top Types to Use in 2023 [Data + Expert Tips] appeared first on ProdSens.live.

Welcome to The Creative, a series that gives content creators actionable advice from professionals in the creator economy. Whether you’re a seasoned creator or just starting out, read The Creative to learn how to grow your platform, improve your content, and stay ahead in the ever-shifting creator landscape. 

As a content creator, I’ve found the most common question other creators in my circle ask is, “How do I collaborate with brands?”

And it makes sense that it’s top of mind for so many creatives since brand deals are among the primary sources of income for podcasters, influencers, YouTubers, streamers, and more.

So, how can a content creator secure a collaboration with a brand? Where do you find these brands? And how do you know you‘re ready? Though I am a creator, I’m more of a hobbyist.

So, I don’t have much personal experience navigating brand deals.

However, I spoke with seasoned creators within HubSpot’s network with experience working with brands. Keep reading to learn everything you need to know about collaborating with brands as a creator, according to the experts.

What is a brand collaboration?

When should you seek brand collaborations?

How to Find Brand Collaborations

What is a brand collaboration?

A brand collaboration is a partnership between a brand and a content creator in which both parties work together to create content that promotes the brand to the creator’s followers.

One of the most common types of brand collaborations, for example, is a partnership.

A brand partnership is when a brand works with an influencer or creator on a joint campaign, offering, or other promotional activity.

An example of a brand partnership would be when makeup and beauty influencer Jackie Aina partnered with makeup brand Anastasia Beverly Hills to release a line of eye shadow palettes.

When should you seek brand collaborations?

I used to think I needed a certain number of followers or that I had to have a plethora of content before even considering working with brands.

So, imagine my surprise when my YouTube channel only had about 100 followers, yet I was already getting brands and businesses in my DMs asking if I wanted to promote their products or services.

It turns out it doesn’t necessarily boil down to high follower counts and viral content when determining the right time to work with brands as creators.

“People can monetize and start collaborating with brands as soon as they have an audience that is useful to the brand,” says Scott D. Clary of the podcast Success Story.

However, Scott warns it’s crucial to understand business practices.

“You have to understand that when you start working with brands, you are working with people who are building their own empires and companies, and — if you don’t have a lot of business experience — it can be overwhelming,” he says.

Scott explains, “They’re going to ask a lot of you. They’re going to negotiate contracts, they’re going to try to ‘win’ in that deal, and they’re going to try to get the best possible bang for their buck with that particular creator.”

So, Scott encourages creators seeking brand deals and partnerships to set themselves up for success by understanding the following:

• The value their content brings
• Their own audience and how said audience will respond to the brand that the creator is working with
• What’s normal in a contract, such as deliverables, reporting timeline, and payment periods

“So, immerse yourself in a variety of different YouTube videos, resources, or blogs based on how to negotiate a good deal and how to serve a brand the best,” he says.

He explains, “It’s almost more important to do the research and understand the mechanics of how to sell advertising as a service first before you jump into bed with brands.”

Scott says it‘s possible to go in without prior research and learn as you go, “but it’ll be painful, and you won‘t be getting good deals, or you won’t be getting paid on time.”

How to Find Brand Collaborations

So, you feel ready to seek collaborators — where do you go?

Leanne Elliot of the podcast Truth, Lies, and Workplace Culture suggests in-person events.

“We have gotten the majority of our content partnerships from being at an event, representing the podcast, talking to guests, interviewing people,” she explains. “From there, either members of their team or they themselves would approach us for collaborations.”

Leanne says it’s crucial to target events you know your desired collaborators will attend, and I can attest to this.

A few years ago, I wanted to create a YouTube video for my channel diving into how the COVID-19 pandemic impacted the voice acting industry.

I knew I wanted to interview an experienced and professional voice actor for the project, so I made a point of attending a local convention and meeting with voice actors in attendance.

That day, I met actress Anairis Quinones, known for voicing various popular anime characters.

I told her about my channel and that I wanted to learn more about her experience in voice acting during a pandemic. Just one week later, we were recording a video for the channel.

Though it wasn’t a brand collaboration, the same logic still applies.

So, you’re a creator and found a brand you want to work with — how do you pitch yourself to that brand?

“You don’t,” says Al Elliot, co-host of the Truth, Lies, and Workplace Culture podcast. “You don’t pitch yourself; you pitch what you’ll give to the collaborator.”

The podcast discusses different aspects of Workplace Culture, and Al says when he meets potential collaborators, he doesn’t approach the person by going on about the podcast.

“I do the opposite,” He explains. “I go, ‘I notice you have some content about workplace culture, but not loads. I think we can collaborate on something really cool where you can tell me about how you see workplace culture.'”

So let the potential collaborator know you want to provide an opportunity for them to discuss their brand, product, or service — emphasize the value your platform brings.

Brand Collaboration Tools

In-person events are just one of the options for connecting with brands. Scott says multiple tools are available to help creators find brands and companies to collaborate with.

For example, if you’re a creator with a newsletter looking for brand partnerships, Scott has a few platforms in mind.

“For newsletter, you could use Paved, Who Sponsors Stuff, SponsorLeads, or Sponsorgap,” he says. “These are all marketplaces where brands that love sponsoring newsletters are already looking for newsletter sponsorships.”

Scott also suggests subscribing to your favorite newsletters, paying attention to what brands are sponsoring them, and running outreach campaigns to the head of marketing for those companies.

If you host a podcast, there are online marketplaces for you to seek brand partnerships as well.

“There’s AdvertiseCast and Gumball,” Scott says. “Or, it‘s going to your favorite podcast that’s in the same niche as you and see who sponsors their podcast, then running outreach campaigns to the head of marketing for that particular company.”

Red Flags to Look For

While it’s great to seek new opportunities as a creator, it’s essential to be selective about who you choose to work with. One way to ensure you’re dealing with the right brands and companies is to be aware of the following red flags.

Only offering Performance- or Affiliate-Based Payments

One potential red flag (or “gray” flag, as Scott puts it) is if a brand doesn’t seem to appreciate the value a content creator brings to a collaboration or partnership.

“Meaning that if you have a great audience, you know who your audience is, and you know a brand will be successful if they work with you, and they’re only asking for a performance-based or an affiliate-based payment — I feel like that is not an ideal scenario for a lot of great content creators,” Scott says.

Performance-based payment means a brand pays a creator based on the results the content generates, such as the click-through rate or social media engagement.

An affiliate-based payment is when a creator is paid after their content directly results in a consumer buying the advertised product or service.

Scott explains, “It’s how some people get started, but I think you have to know your worth as a creator, and you can’t pay your bills with potential future revenue.”

Instead, a better option would be for creators to have their own rate or negotiate their contracts to guarantee a proper, consistent, and fair income.

Offering to Pay in Free Products or Equity

“That’s even more of a red flag,” Scott says. “When brands say, ‘We’re just going to give you free products, and we expect so many posts.'”

The obvious issue with only getting free products as payment is that these items can’t pay your bills in the long run, no matter how nice they are.

Like, thanks for the free shampoos, but I can’t exactly appease my landlord with these when it’s the first of the month.

According to Scott, equity compensation is another payment method to be wary of.

If you’re unsure what equity compensation is, it’s when a brand offers a creator a payment in the form of things like restricted stock or a performance share.

This form of payment is especially problematic because there‘s a good chance the equity isn’t really worth much and may not result in any cash down the line.

“The equity the brand is giving you could be so diluted, and the valuation of the company so overvalued, that you’re actually probably never going to walk away with any money,” Scott warns.

The Brand is Too Controlling

A brand collaboration is supposed to be exactly that — a collaborative effort, meaning both parties work together for a mutually beneficial outcome.

For that to happen, the brand you choose to work with must respect the integrity of your work and not micromanage your process.

Not only can an overly controlling brand negatively impact your experience, but it can also hurt your content and damage your audience’s trust.

“We see that a lot in terms of YouTubers and influences who have gotten a lot of heat from the press and the YouTube community for promoting products and using scripts,” Leanne explains. “You see the same four or five influencers saying the same things about the same products, and it’s clearly been scripted by that brand.”

She says, “Particularly from a podcast medium, we need to be very protective of our content.”

This is because podcasts are one of the most trusted mediums for information, especially among Gen Z. In fact, 47% of the Gen-Z online population in the U.S. are monthly podcast listeners.

And 64% of Gen-Zers in the UK say podcasts are more trustworthy than other media.

“We need to be extra careful, as podcast creators, to ensure integrity is really high,” Leanne says. “So a red for me would be anybody who wants more control, or more say in my content than I feel comfortable with.”

Green Flags to Look For

You want to work with brands who are transparent and honest about their payment methods and who will enter into a fair and equitable contract with you.

Another good sign a brand is worth working with is that they’ve worked with creators in the past and have a sense of how brand collaboration works.

Finally, you want to work with brands who will respect you as a creator and trust you to bring their vision to life rather than micromanaging your work or having too much input on how you communicate their brand to your audience.

Over the years, I‘ve seen many content creators get excited about the possibility of securing a brand collaboration or partnership.

Still, creators must know what to expect before working with companies; otherwise, there’s a strong possibility they‘ll get stuck with a deal that doesn’t yield a high return on investment.

So, do your research, trust yourself as a creator, and make sure to work with brands who take your work seriously and will compensate you fairly.

The post How to Collaborate with Brands as a Content Creator [Expert Tips] appeared first on ProdSens.live.

Currently the vast majority of data warehouses employ SQL to process data. Following decades of development, SQL has become the standard language in the database world, and amassed a large user population, so it is normal to support SQL for data warehouses. However, in the context of contemporary big data, as the business complexity keeps increasing, the abilities of SQL seem to be increasingly inadequate in the data warehouse scenario where computation is the primary task. A typical manifestation is that some data warehouses begin to integrate non-SQL languages like Python. Not to mention whether the two languages with very different styles can be well integrated and complementary, just judging from this trend, it is clear that the industry is doubtful of the abilities of SQL.

In this article, we will present a non-SQL-based data warehouse “esProc”. Since esProc does not use SQL as query language (SPL instead), we can regard it as a new type of data warehouse temporarily.

Why doesn’t esProc use SQL?

To answer this question, we need to ascertain the reason why the data warehouses still introduce Python even when SQL is already available, and what problems does it want to solve?

We know that SQL does not provide good support for procedural computation. Even with CTE syntax, it is still very complicated to describe complex calculations in SQL, and often requires nesting multiple layers of code and associating repeatedly. In addition, the dataset of SQL is unordered, and SQL is very bad at ordered calculation. When handling order-related operations, coding in SQL is often cumbersome or even impossible to implement. The language characteristic of SQL itself determines that it is not good at implementing certain complex calculations., yet such calculations are common in the data analysis scenarios of data warehouse. For example, it is difficult to code in SQL when performing a funnel analysis for an e-commerce company (calculating the user churn rate of every step such as page browsing, adding to cart, placing order, and paying). In contrast, for the scenarios involving multiple sequential steps and repeated use of result, it is much easier to implement in a language (like Python) that supports stepwise and ordered calculations.

In other words, SQL lacks adequate abilities.

However, introducing third-party languages like Python to make up for the lack of abilities will make the technology stack complex. Even if we don’t consider whether Python can make up for the lack, and nor do we consider whether the two languages can be well incorporated, the rise in system complexity brought about by multiple different-style technologies alone will inevitably lead to high development and O&M costs.

In addition to the lack of abilities, the SQL-based relational database has the problem of closedness.

Since the main function of database is transaction processing (TP), it requires many constraints to ensure data consistency, etc. For example, only the data that meet criteria can be loaded into database; only the data inside the database can be processed, which is what we called the closedness. The data warehouse is developed based on database, and inherits the closedness of database.

Closeness is very important for TP business but, it is meaningless and even very disadvantageous for AP business that mainly focuses on analysis and calculation. The closedness requires that the data can only be used after being loaded into the database, this will result in the inability to combine and calculate the data from multiple databases at will, thus greatly limiting the application scenarios of data warehouse.

In addition, the data sources of modern data application are diverse. In addition to different databases, we often face a variety of data sources and data types. Since the closed SQL-based database cannot compute the data outside the database, it has to import the data before computing, resulting in the addition of an ETL action. This action not only increases the workload of programmers and the burden of database, but losses the real-timeness of data. Usually, the data outside the database have irregular formats, and it is not easy to load them into databases having strong constraints and, even ETL action is performed, it first needs to load the raw data into database in order to utilize database’s computing ability. As a result, ETL is changed to ELT, which increases the burden of database.

Moreover, the closedness makes it inconvenient for users to freely utilize the method of trading space for time. We know that the storage resource is much cheaper than the computing resource. If we redundantly store data in multiple ways for different computing objectives, we may obtain better query experience. However, SQL needs to store the data to tables. Creating too many tables will make the metadata bigger, resulting in a significant increase in O&M costs and, too many tables will also bring capacity and performance issues to the data warehouse, facing the scaling pressure. Many large organizations store thousands of intermediate tables in their central data warehouses. These tables have been accumulated for years, but have to be retained out of an abundance of caution, thus causing immense burden on the capacity, performance, operation and maintenance of the database.

SQL doesn’t do a good job either in performance

As we know, the execution efficiency of SQL depends on the optimization degree of database optimization engine, and a good database will choose more efficient execution path according to the computing objective of SQL (rather than its literally expressed logic). However, such auto-optimization mechanism works only for simple calculations. Once the calculation becomes slightly more complex, the engine will not work, and the code has to be executed according its literally expressed logic, resulting in a sharp decline in performance. For example, for the funnel analysis task mentioned above, someone wrote a three-step funnel calculation code in SQL and executed in the database, yet the speed was too slow to be feasible. We believe that you must have often encountered cases where SQL performs poorly in actual business and, it is common to see that it takes a couple of hours to run SQL codes in many batch job scenarios, all of which are caused by the low performance of SQL.

The lack of abilities, closedness (results in heavy in use) and low performance are the main problems the SQL-based data warehouse is facing.

Introducing Python based on SQL cannot solve problems either. In addition to high use and O&M costs caused by complex technology stack mentioned above, Python also cannot achieve high performance.

Python itself doesn’t provide good support for big data computation, and does not offer corresponding external storage computation types (such as cursor) for the calculation of data exceeding memory capacity, making it exceptionally complicated to process big data. Moreover, Python does not support true multi-thread parallel processing. The parallel processing of Python is fake, which is actually the serial processing for CPU, or even slower than serial processing, making it difficult to leverage the advantages of modern multi-core CPU.

More importantly, Python needs to calculate based on SQL database tables, but these tables (storage) cannot be intervened by the outside world as they are private to the database. Many high-performance algorithms, however, need to organize the data based on computing objective. For example, the efficient ordered merge algorithm can be utilized if the data are sorted by associated fields. Unfortunately, the high-performance algorithms cannot be utilized due to the failure to intervening the storage, and it naturally fails to ensure the performance. In addition, when Python reads the data of database, it will involve IO costs, which will also lead to low computing performance.

It seems that to solve the problems of SQL, we have to abandon SQL.

It is a fact that non-SQL computing technologies have always been present, and the typical representative is Spark. When Spark was born, Scala was used as the programming language and, relying on Spark’s large-scale distributed computing ability, there was a great tendency to replace SQL. Unfortunately, however, as Spark was used more deeply, we realized that Spark does not have the ability to replace SQL (because the implementation process is too cumbersome, and the performance is low). In addition, due to the difficulty to use Scala, the programmers have to resort to SQL again.

Next let’s take a closer look at the abilities of esProc to see what the differences are.

esProc SPL

The formal language of esProc-based data warehouse is SPL, not the SQL commonly used in the industry. The reason for abandoning SQL is that SQL has many problems, such as the lack of abilities, closedness, and low performance, while SPL can effectively solve these problems. Here below are some advantages of SPL.

Complete capability

First, SPL naturally supports procedural computation.

Procedural computation can effectively reduce the implementation difficulty of complex business. For the same code of 100 lines, the complexities of writing it as 100 statements and one statement are completely different. Although CTE syntax and stored procedure make SQL have the procedural computing ability to a certain extent, it is far from enough. In contrast, SPL naturally supports procedural computation, allowing us to divide complex calculation into multiple steps to reduce the implementation difficulty.

Second, SPL provides richer data types and algorithms.

Compared to SQL that doesn’t offer explicit record data type (SQL will treat a single record as a temporary table with only one record, i.e., a single-member set), SPL provides a specialized structured data object: table sequence and provides rich computing libraries based on the table sequence, thereby making SPL have complete and simple structured data processing ability.

Here below are part of common calculation codes written in SPL:

Orders.sort(Amount) // sort
Orders.select(Amount*Quantity>3000 && like(Client,"*S*")) // filter
Orders.groups(Client; sum(Amount)) // group
Orders.id(Client) // distinct
join(Orders:o,SellerId ; Employees:e,EId) // join

By means of the procedural computation and table sequence, SPL can implement more calculations. For example, SPL supports ordered operation more directly and thoroughly. For the grouping operation, SPL can retain the grouped subset, i.e., the set of sets, which makes it convenient to perform further operation on the grouped result. In contrast, SQL does not provide explicit set data type, and cannot return the data types such as set of sets. Since SQL cannot implement independent grouping, grouping and aggregating have to be bound as a whole.

In addition, SPL has a new understanding on aggregation operation. In addition to common single value like SUM, COUNT, MAX and MIN, the aggregation result can be a set. For example, SPL regards the common TOPN as an aggregation calculation like SUM and COUNT, which can be performed either on a whole set or grouped subsets.

In fact, SPL has many other features, making it more complete than SQL and richer than Python. For example, the discreteness allows the records that make up a data table to exist dissociatively and be computed repeatedly; the universal set supports the set composed of any data, and allows such set to participate in computation; the join operation distinguishes three different types of joins, allowing us to choose an appropriate one according to actual situation…

With complete computing capabilities, not only is it simple to code, but it also eliminates the need to resort to other computing capabilities. Therefore, the technology stack is simple, and all problems can be solved within one system.

Open system

Unlike the SQL-based database that requires loading data into database before calculation (closedness), SPL can directly calculate when facing diverse data sources, and hence it has good openness.

SPL does not have the concept of “warehouse” of traditional data warehouses, nor does it have the concept of metadata, let alone constraints. Any accessible data source can be regarded as the data of esProc and can be calculated directly. Importing the data into database is not required before calculation, and exporting the data out of database deliberately is also not required after calculation, as the result can be written to target data source through its interface.

SPL encapsulates access interfaces for common data sources such as various relational databases (JDBC data source), MongoDB, HBase, HDFS, HTTP/Restful, SalesForces and SAP BW. Logically, these data sources have basically the same status, and can be calculated separately or in a mixed way after being accessed, and the only difference is that different data sources have different access interfaces, and different interfaces have different performance.

Efficient file storage

In terms of data storage, SPL differs greatly from traditional data warehouses.

SPL has no metadata. The data of SPL is directly stored in files, and any type of open-format file can be processed. In order to ensure computing performance, SPL also designs a specialized binary file format.

Currently, SPL provides two file types: bin file and composite table. The bin file adopts the compression technology (faster reading due to less space occupation), and stores the data types (faster reading as a result of avoiding parsing data type). Since the bin file supports the double increment segmentation mechanism that can append data, it is easy to implement parallel computing by utilizing the segmentation strategy, and the computing performance is ensured. The composite table supports the columnar storage, which has great advantage when the number of columns (fields) involved in calculation is small. In addition, the composite table not only supports the double increment segmentation mechanism, but adopts the index technology, allowing us to utilize the advantage of columnar storage, and improve the performance more easily through parallel computing.

Since the binding of storage and computation is eliminated, it is easy to implement the separation of storage and computation and then implement elastic computing, and it also makes the cloud computing easier.

In addition, the cost of file storage is lower. In AP computing scenario, users can flexibly design a space-time trade-off scheme, which is nothing but storing a few more files. Even if the number of redundant files reaches up to ten thousand (it is easy for contemporary file systems to handle the data of such a scale), there isn’t any burden. Also, it is simple to manage data files by category under the file system’s tree structure. And the O&M costs are lower.

For more information, visit: Data warehouse running on file system

High performance

Based on flexible file storage, we can design the data organization form (storage) flexibly according to computing objective to achieve high performance. In addition to high-performance storage, SPL provides many high-performance computing mechanisms and algorithms for big data.

In order to cope with the big data computing scenario where the amount of data exceeds memory capacity, SPL offers cursor computing method.

=file("orders.txt").cursor@t(area,amount).groups(area;sum(amount):amount)

Moreover, SPL provides parallel computing support for both in-memory and external storage calculations. By adding just one @m option, parallel computing can be implemented and the advantages of multi-core CPU can be fully utilized, which is very convenient.

=file("orders.txt").cursor@tm(area,amount;4).groups(area;sum(amount):amount)

n addition to cursor computing and the parallel computing, SPL offers many built-in high-performance algorithms. For example, after SPL treats the TOPN as an ordinary aggregation operation, sorting action is avoided in the corresponding statement, so the execution is more efficient.

Similarly, SPL provides many such high-performance algorithms, including:

• In-memory computing: binary search, sequence number positioning, position index, hash index, multi-layer sequence number positioning…
• External storage search: binary search, hash index, sorting index, index-with-values, full-text retrieval…
• Traversal computing: delayed cursor, multipurpose traversal, parallel multi-cursor, ordered grouping and aggregating, sequence number grouping…
• Foreign key association: foreign key addressization, foreign key sequence-numberization, index reuse, aligned sequence, one-side partitioning…
• Merge and join: ordered merging, merge by segment, association positioning, attached table…
• Multidimensional analysis: partial pre-aggregation, time period pre-aggregation, redundant sorting, boolean dimension sequence, tag bit dimension…
• Cluster computing: cluster multi-zone composite table, duplicate dimension table, segmented dimension table, redundancy-pattern fault tolerance and spare-wheel-pattern fault tolerance, load balancing…

With the support of high-performance file storage and algorithms, esProc often achieves a performance improvement of several times to dozens of times, and even thousands of times in some cases, compared to traditional SQL-based data warehouses in practice.

Can we draw a conclusion that SPL have no disadvantages compared to SQL?

Of course not, because nothing is perfect in this world.

After decades of development, many SQL-based databases have owned powerful optimization engine. For simple operations that are suited to be implemented in SQL, the optimization engine can optimize the slow statements written by ordinary programmers and achieve better performance. In this sense, the requirements for programmers are relatively low. Certain scenarios, such as multidimensional analysis, have been optimized for years, some SQL engines can also handle them very well and obtain extreme performance.

In contrast, SPL did little in automatic optimization., and depends almost entirely on programmers to write low-complexity code to achieve high performance. In this case, programmers need receive some training to familiarize themselves with SPL’s philosophy and library functions before getting started with SPL. Besides, SPL is written in Java, which brings some benefits such as good compatibility, strong migration, and easy to adapt to the cloud environment. However, the CPU and memory resources cannot be fully utilized due to JVM limitations. For some simple scenarios, SPL is still not as good as a fully optimized SQL engine in terms of performance.

In conclusion, SQL is not the only option for data warehouse, we have a better alternative – SPL.

Original link :Data warehouse without using SQL
Our Github link : Github link SPL

If you have any other questions, please leave a message in the comment area or send me an email: viv.esprocspl@gmail.com.

The post Data warehouse without using SQL appeared first on ProdSens.live.

The journey of taking an open-source artificial intelligence (AI) model from a laboratory setting to real-world implementation can seem daunting. However, with the right understanding and approach, this transition becomes a manageable task.

This blog post aims to serve as a compass on this technical adventure. We’ll demystify key concepts, and delve into practical steps for implementing anomaly detection models effectively in real-time scenarios.

Let’s dive in and see how open-source models can be implemented in production, bridging the gap between research and practical applications.

Understanding Unsupervised Anomaly Detection

Unsupervised anomaly detection is a machine learning technique that uncovers unusual patterns or outliers in data, without any prior training on what these anomalies might look like. In the context of images, this means identifying areas within the image that deviate significantly from what’s considered ‘normal’.

Implementing that in real-time involves using deep-learning models which can rapidly process incoming visual data, detect irregularities in a matter of milliseconds, and respond accordingly. It’s like having a vigilant digital watchdog capable of recognizing anything out-of-the-ordinary at high speed.

With open-source models, you have access to this technology as well as the collective wisdom of researchers and developers worldwide who continually refine these tools for better performance.

Role and Importance of Anomaly Detection in Images

Anomaly detection in images plays a crucial role in numerous fields, ranging from healthcare to security. In healthcare, it can aid in identifying abnormal structures or changes in medical imagery like X-Rays or MRI scans, potentially flagging early signs of diseases. In security applications such as surveillance systems, it can help detect unusual activities or objects within the monitored area.

The importance of anomaly detection also extends to quality control in manufacturing, where it can spot defects on assembly lines avoiding costly recalls and, hopefully, ensuring customer satisfaction.

Real-time implementation of open-source models for this purpose allows these sectors to react quickly to anomalies and make informed decisions instantly.

Anomalib: A Deep Learning Library for Anomaly Detection in Images

Anomalib is an open-source library for unsupervised anomaly detection in images. It offers a collection of state-of-the-art models that can be trained on your specific images.

Example of an anomaly detected with Anomalib

For achieving the best training results, it is advised to obtain a suitable quantity of images that are free from any abnormalities. It is preferable to have a few hundred images for this purpose. Furthermore, in order to perform testing and validation, it is recommended to acquire a few images that do include anomalies as well.

If your images have anomalies, you’ll need to make a mask highlighting these areas. You don’t need a sophisticated tool for that. For example with GIMP, it’s as simple as:

• Drag and drop your original image (e.g., “001.png”) into GIMP.
• Go to the Layer menu and select New Layer.
• With the new layer selected, use the Pencil tool set to white to mark the anomalies.
• Use the Bucket Fill tool to color the unmarked areas in black.
• Save the mask using File -> Export As, naming it “001_mask.png”.

Keep in mind that masks are for testing, not training. Thus, only a few anomaly images are required for this.

Afterward, you can train one of the Anomalib models and test its performance under simulated conditions. I would recommend EfficientAd or FastFlow for real-time applications, as they are significantly faster than other models as of October 2023. Additionally, you can explore Anomaly Detection on Paper With Code to find the ideal model for your specific use case.

The easiest way to get started with Anomalib, is to clone their repository from Github and use the train script as follows:

python tools/train.py --config  --model 

Sample config files are available in the repo, and it lets you set the paths of the folders containing your pictures for training and testing. Then, once your model is trained and validated, you can use the inference script to test it under simulated conditions on a single image or a folder of images. For example, with PyTorch, you can run the inference script as follows:

python tools/inference/torch_inference.py 
    --weights results/your/model.pt 
    --input your/image.png 
    --output results

This will generate a visualization that highlights the areas of the image that the model has identified as anomalies. You can set the flag --visualization_mode between full and simple to change the visualization mode. Select full to view the original image with its mask, heatmap, and segmentation.

From Lab to Live: Implementing Your Models With ONNX or OpenVINO

Once your model has been trained and validated using Anomalib, the next step is to prepare it for real-time implementation. This is where ONNX (Open Neural Network Exchange) or OpenVINO (Open Visual Inference and Neural network Optimization) comes into play.

ONNX offers a standardized platform that allows you to export your trained model into a format that can be easily implemented and run in various environments.

OpenVINO is another toolkit but developed by Intel. Its primary purpose is to facilitate the rapid deployment of deep learning models for inference.

To convert your model, you will need to add this configuration to your config.yaml file to export your model to ONNX or OpenVINO format after training.

optimization:
  export_mode: < openvino or onnx >

Once converted, the model can be embedded in your application with ONNX Runtime or OpenVINO Inference Engine respectively.

Deploying New Models in Shadow Mode

Deploying in shadow mode is a crucial step when introducing a new model to production. This deployment strategy involves running the new model alongside your existing system without directly influencing the output, essentially running in the “shadow”.

It’s like having a student trying to outperform the master. During this phase, both models process each image concurrently but only results from your current system are utilized, while outputs of the new model are monitored and compared for any discrepancies.

This allows you to assess how well the new model performs under real-world conditions without risking any impact on your operations if it doesn’t perform as expected. It also provides an opportunity to fine-tune parameters or retrain the model with more specific data based on its performance during shadow operation.

Once satisfied with its performance and reliability, you can then switch over from your old system to this newly deployed model.

ReductStore: Storing AI Labels and Models at the Edge with a Time-Series Database for Blob Data

ReductStore is an innovative time-series database designed specifically for managing Blob data, making it ideal for our needs in real-time unsupervised anomaly detection. The true strength of ReductStore lies in its ability to store not just raw data but also AI labels within the metadata and models at the edge.

To better visualize how ReductStore can integrate with your machine learning workflow, from data capture to inference, consider the following diagram. It provides an overview of how we can make the most out of AI labels and models stationed at the edge.

Diagram illustrating the flow of data capture, storage, inference, and training with ReductStore.

AI labels indicate the outcomes of your model’s evaluation on each picture, like the presence or absence of an anomaly in our case. By storing these labels alongside your images in ReductStore, you streamline your system’s workflow and make the whole process simpler.

Furthermore, annotated images are manually labeled or validated by a human operator, and then stored in ReductStore. This annotation can be the actual label inferred by the model which is then validated by a human operator, or it can be more complex, like a mask highlighting the areas of the image that the model should identify as anomalies. This allows for continuous improvement of your model’s performance over time.

Meanwhile, keeping models at the edge means deploying your trained models directly onto end-user devices or closer to where data is generated. This method cuts down on latency issues since you don’t need to transmit large volumes of image data over networks; instead, you analyze it right where it’s collected.

Conclusion

In conclusion, implementing open-source models for real-time unsupervised anomaly detection in images is a multi-step process that involves transitioning from lab to live.

By selecting the right model, testing under simulated conditions, integrating it into your existing system and regularly monitoring its performance, you can effectively detect anomalies in image data.

Utilizing tools like Anomalib and ReductStore help to facilitate this process by providing robust models and storage solutions respectively. Deploying new models in shadow mode further minimizes risk during the transition phase ensuring that your operations remain unaffected while introducing new models into your system.

Stay tuned for more advancements in the field of unsupervised anomaly detection and the continuous evolution of tools and techniques that will make the process even more streamlined and efficient in the future!

The post From Lab to Live: Implementing Open-Source AI Models for Real-Time Unsupervised Anomaly Detection in Images appeared first on ProdSens.live.

I believe we all entered the field of programming for various reasons. It could be to earn a living, pursue a sought-after career, or simply because we love building stuff. Whatever the reason, we’re here.

However, if I’m being perfectly honest, while all these are good reasons, none would provide me with more than the bare minimum level of happiness at what I do almost every day for many hours.

Yet, while I had my fair share of miserable days on the job like everyone else, more often than not, I am truly eager to do my thing at work.

So today, while I was running, I found myself in a bit of an introspective mood and wondered what is my personal Why? Why do I still love programming so much after all these years.

I always knew it had something to do with people. Seeing someone using something I wrote and maybe even liking it never ceases to give me a kick. But I felt there was a deeper desire.

After a little back and forth with myself I reduced it to something that felt really true for me: to reduce the suffering of someone else. Okay, I know it sounds a bit overly dramatic, but hear me out here for a minute.

Our profession is riddled with sharp objects we all occasionally bump into. People much smarter than myself say that it takes a very long time to even begin to master it, there is formidable math and sophisticated algorithms lurking at every corner, then there are new languages, tools, frameworks and paradigms jumping on us every other day that threaten to undermine everything we’ve learned for the past however many years.

So when I get to brighten someone’s day through my work in even the smallest way, damn it it feels good.

It could be a user that with the help of something I wrote suddenly feels much more productive, or it could be as “small” as assisting a colleague by showing them how to use some tool that I take for granted, but is life-changing for them.

That’s why I relish at the opportunity to spend that extra hour at making my API just a tiny bit simpler, or clean up and refactor some messy code, or write that extra page of documentation or test. Because someone (including myself) will experience just a tiny bit less frustration and pain down the road when they try to use it.

And whenever I get to see it first hand it gives me the energy to wake up the next day and do it all over again.

So now I’d like to invite you to find your personal Why, and to please share it with us.

The post What is your Why? appeared first on ProdSens.live.

Introduction:

In the realm of software development, effective communication is paramount to the success of any project. Among the crucial elements of communication are tasks and software requirements. These specifications outline the foundation for building a software application and serve as a roadmap for developers.

However, the question arises: should the description of tasks and software requirements be the same for junior, mid, and senior developers? In this blog post, we will explore the nuances of this topic and delve into the benefits of tailoring software requirements to meet the diverse needs of developers at different experience levels.

Understanding Developer Levels:

• Junior Developers
  are at the early stages of their careers, eager to learn and grow. They rely on precise instructions to avoid misunderstandings and build their technical expertise.

• Mid-Level Developers
  possess a solid grasp of both technical skills and business requirements. They can translate business needs into technical solutions.

• Senior Developers
  bring extensive experience and expertise to the table. They thrive on understanding the big picture and aligning technical solutions with strategic goals.

Tailoring Software Requirements:

Clarity and Detail for Juniors:

They rely on precise instructions to avoid misunderstandings and build their technical expertise. Well-defined requirements enable them to develop confidence and work with more independence under the guidance of their senior counterparts.
Employing clear language, well-defined use cases, and providing visuals can greatly facilitate their understanding, also including detailed acceptance criteria helps them identify project success criteria and build robust solutions.

Balancing Clarity and Flexibility for Mids:

Mid-level developers require a balance between well-defined requirements and room for interpretation. Providing a clear problem statement and outlining the expected outcome allows them to leverage their problem-solving abilities effectively. Allowing some flexibility within the specifications encourages mid-level developers to contribute creatively and think critically about the project.

High-Level Guidance for Seniors:

For senior developers, high-level requirements with a focus on business and technical objectives are most beneficial. They appreciate the opportunity to contribute their expertise to the decision-making process. Presenting the project’s goals and constraints allows senior developers to devise elegant solutions, tailored to the unique challenges posed by the project.

Example

Here are examples of software requirements tailored to the perspectives of a junior, mid-level, and senior developer.

Example of Software Requirements for Junior Developer:

Feature: User Registration
Description: The application should allow users to register for an account.

Acceptance Criteria:

• Implementing the front-end registration form with the required fields (email and password)

• Writing basic validation for the email format and password length. The user must provide a valid email address and a password with at least 8 characters.

• Show a success message upon successful registration and redirect the user to the login page.

• If the user attempts to register with an existing email, an error message should be displayed.

Example of Software Requirements for Mid-Level Developer:

Feature: Shopping Cart
Description: The application should include a shopping cart functionality.

Acceptance Criteria:

• The user should be able to browse the product catalog and add items to the cart by clicking the Add to Cart button

• The user can view the cart by clicking the cart icon, which opens a dropdown displaying the cart items and total cost.

• When the user proceeds to checkout, they should be directed to the payment page to enter their payment details.

• After successful payment, the user should receive an order confirmation email.

Example of Software Requirements for Senior Developer:

Feature: Recommendation Engine
Description: The application should include a recommendation engine that provides personalized product recommendations to users based on their browsing and purchase history.

Acceptance Criteria:

• The recommendation engine should analyze user behavior, including past purchases and product views, to create user profiles.

• Based on the user profile, the engine should display personalized product recommendations on the homepage and product pages

• The engine should prioritize recommendations based on relevance and user preferences.

Collaboration and Communication:

The key to tailoring software requirements effectively lies in open communication and collaboration within the development team. Regular meetings, stand-ups, and brainstorming sessions allow developers of all levels to share their perspectives, seek clarification, and contribute ideas. This fosters a positive working environment where each team member’s strengths can shine.

Conclusion:

In conclusion, while the core purpose of software requirements remains constant—to guide the development process—it is crucial to tailor their level of detail and specificity to the different needs of junior, mid, and senior developers. Clarity empowers juniors, flexibility engages mid-level developers, and high-level guidance allows seniors to contribute their expertise. By adapting software requirements to match the experience levels of developers, we promote a collaborative and inclusive development environment, ultimately leading to successful software projects. Remember, it’s not about making the description the same for all levels, but about making it fit for the purpose of each level.

The post Tailoring tasks and software requirements: Addressing the Needs of Junior, Mid, and Senior Developers appeared first on ProdSens.live.

In this week’s episode of Dev Interrupted, we’re focusing on the increasingly valuable role of DevRels. Francesco Ciulla, Developer Advocate at the open-source daily.dev community – which has more than 100,000 daily active users – joins us for a DevRel deep dive.

Listen as Francesco explains how a career change in his thirties set him on a path towards becoming a developer, being hired by the European Space Agency and, eventually, landing a role as a developer advocate, crediting much of his success on his ability to leverage social media to advance his career.

Outside of his personal story, Francesco shares his thoughts on connecting with devs, why YouTube is such a powerful platform and settles the debate on the kind of content developers are most interested in.

Episode Highlights:

• (0:00)
• (3:05) Francesco’s career change to programming
• (10:15) How to leverage social media
• (16:10) Best ways to connect with devs as a DevRel
• (22:28) Challenges of being a DevRel
• (25:42) What platforms should DevRels be using?
• (30:07) Community building on YouTube
• (33:12) Technical vs non-technical content

Read the full episode transcript

While you’re here, check out this video from our YouTube channel, and be sure to like and subscribe when you do!

A 3-part Summer Workshop Series for Engineering Executives

Engineering executives, register now for LinearB’s 3-part workshop series designed to improve your team’s business outcomes. Learn the three essential steps used by elite software engineering organizations to decrease cycle time by 47% on average and deliver better results: Benchmark, Automate, and Improve.

Don’t miss this opportunity to take your team to the next level – save your seat today.

The post The Art of Landing a DevRel Role with daily.dev appeared first on ProdSens.live.

What is Open Source?

• Open-source software is software whose source code is publicly available and can be modified and distributed by anyone.
• The open-source model is based on collaboration and community-driven development, which allows for faster innovation and bug fixing.
• Engineers can take advantage of the vast amount of open-source software libraries and tools available to them, which can save them time and resources when working on projects.
• Engineers can also contribute to open-source projects, which can help them gain experience and improve their skills.
• Using and contributing to open-source projects can also help engineers to build a strong professional network and reputation in the industry.
• It could be a great way to learn and improve your coding skills by studying the code of experienced developers.

Why Open-Source and why you?
There are several reasons why students should consider participating in open-source internship programs like Google Summer of Code (GSOC):

1. Hands-on experience: Participating in open-source projects through programs like GSOC allows students to gain real-world experience working on software development projects.
2. Career development: Participating in open-source projects can help students to build a strong portfolio and demonstrate their skills to potential employers.
3. Networking: Open-source projects provide a platform for students to connect and collaborate with experienced developers, which can help them to establish a professional network in the industry.
4. Learning opportunities: Participating in open-source projects can expose students to new technologies and programming languages, which can help them to expand their knowledge and skill set.
5. Giving back to the community: Open-source projects allow students to contribute to the development of software that is used by millions of people worldwide, which can be a rewarding experience.
6. GSoC is a great way to improve your coding skills and gain experience by working on real-world projects under the mentorship of experienced developers. It also provides a stipend, so it’s a great way to earn while learning.
7. Open-source projects can give you a chance to work on cutting-edge technologies and work on challenging problems.

Overall, participating in open-source internship programs like GSOC can provide a wealth of learning and career development opportunities for students.

Here I have made a list of all Open Source Internship Programs to vest yourself into the world of opensource program, get loads of practice, make your resume strong, a fantastic stipend and most importantly to have fun and build something useful and credible :

Google Summer of Code(GSoC): https://summerofcode.withgoogle.com/

Girlscript Summer of Code: https://gssoc.girlscript.tech/

Hacktoberfest: https://hacktoberfest.com/

Reinforcement Learning Open Source Fest: https://www.microsoft.com/en-us/research/academic-program/rl-open-source-fest/

Linux Foundation Mentorship Program (LFX): https://lfx.linuxfoundation.org/tools/mentorship/

MLH Fellowship: https://fellowship.mlh.io/

Google Season of Docs (GSoD): https://developers.google.com/season-of-docs

Outreachy: https://www.outreachy.org/

Season of KDE: https://season.kde.org/

Free Software Foundation (FSF) Internship: https://www.fsf.org/volunteer/internships

Linux Kernel Mentorship Program: https://wiki.linuxfoundation.org/lkmp

Linux Foundation Networking (LFN) Mentorship Program: https://wiki.lfnetworking.org/display/LN/LFN+Mentorship+Program

FOSSASIA Codeheat: https://codeheat.org/

FOSSASIA Internship Program: https://fossasiataipei.github.io/fossasia-cht/apply/

Red Hat Open Source Contest: https://research.redhat.com/red-hat-open-source-contest/

Segment Open Fellowship: https://segment.com/opensource/

Open Summer of Code: https://osoc.be/

Open Mainframe Project Mentorship Program: https://www.openmainframeproject.org/all-projects/mentorship-program

CNCF Mentoring Initiatives: https://github.com/cncf/mentoring

X.Org Endless Vacation of Code (EVoC): https://www.x.org/wiki/XorgEVoC/

Hyperledger Mentorship Program: https://wiki.hyperledger.org/display/INTERN

Julia Seasons of Contributions (JSoC): https://julialangblogmirror.netlify.app/jsoc/

Summer of Haskell: https://summer.haskell.org/

24 Pull Requests: https://24pullrequests.com/about

Summer of Bitcoin: https://www.summerofbitcoin.org/

New programs are introduced throughout the year and many programs can be added and you can just drop a quick comment for me in the comment section and I will add it here.

The post All Open Source Internship Programs at One Place appeared first on ProdSens.live.

When I was a teenager, our local telephone company introduced a new service – the premium phone calls (AKA 1-900 numbers). The fun part was that we discovered a workaround to these charges by dialing the sequential local numbers which these 1-900 numbers would redirect to. If the “support number” for the 1-900 was 555-555 we would dial every number between 555-455 and 555-655 until we hit the jackpot…

Hours were spent dialing these numbers, leading us to make numerous calls for free. This attack is still prevalent today, and it’s called Insecure Direct Object References (IDOR).

IDOR

In the digital world, IDOR is similar to our teen exploits. It means trying various ID numbers in sequence until we find the right one. A few years ago, a social network named Parler, which listed users by a sequential numeric ID, fell victim to this type of attack when a user was able to request and download the full list of users on that network.

E.g. their URLs looked like: https://site.com/viewUser?id=999

All a person needs to do is loop over valid sequential numbers and send the request to get the user information of everyone on that site. This is trivial and can be accomplished by anyone with relatively low technical skills.

To avoid such an attack, it is advised not to expose guessable or sequential numeric IDs to the end users. While UUID might seem long, it offers a more secure alternative. Additionally, request checking should be implemented. If a user is requesting information about a user they aren’t connected to, that request should be blocked. Other effective mitigations include setting request quotas and delays between requests to prevent a massive data grab.

I won’t go into these since they are typically implemented in the API gateway layer during provisioning. You can write this in code but it’s a challenging task as you might have many endpoints with a great deal of complexity. The rule of thumb is to write as little code as you possibly can, more code means more bugs and a wider attack surface for a malicious hacker.

Vulnerabilities and Exploits

A crucial term in application security is vulnerability. It’s a weakness or bug that can be likened to a hole in the fence surrounding your house. These vulnerabilities can reside in your code, libraries, Java itself, the operating system, or even physical hardware. However, not every vulnerability is exploitable. Just like a hole in your fence may not necessarily grant access to your house, vulnerabilities don’t always mean your code can be hacked. Our aim is to plug as many holes as possible to make the task of exploiting our system more difficult.

I know the onion metaphor is tired by now but for security it makes a lot of sense. We need to enforce security at every layer. In the Log4Shell exploit that was exposed last year we had a major zero-day vulnerability. A zero-day vulnerability is a newly discovered vulnerability that no one knew about before, like a new hole in the fence.

The Log4Shell vulnerability relied on people logging information without validating it first. This was a bad practice before the vulnerability was known. If you used a Log4J version that had that vulnerability, but sanitized your data. You would have been safe despite that vulnerability.

SQL Injection

SQL injection involves building your own queries by concatenating a query string manually. Let’s look at vulnerable SQL like this:

String sql = "SELECT * from Users WHERE id = " + id;

Considering the sample URL we used before we could request a URL like this: https://site.com/viewUser?id=1 OR true=true.

This URL would result in an attacker fetching all the users as the condition will become:

SELECT * from Users WHERE id = 1 OR true=true

Which is always true. This is a relatively tame outcome. SQL statements can be chained to drop tables deleting the entire database. A solution to this is using the prepared statement syntax, where the implementation treats all the content as a string. This prevents the SQL keywords from being exploited e.g.:

PreparedStatement sql = connection.prepareStatement("SELECT * from Users WHERE id = ?");
sql.setString(1, id);

In this situation when we set the value for the id it will treat it as a string even if there are SQL keywords or special characters. Using APIs like JPA (Spring Data, Hibernate, etc.) will also protect you from SQL injection when using similar APIs.

Serialization

Java serialization is another common vulnerability. The lesson here is to avoid using serialization or requiring it, and instead running your app with a filter that blocks certain types of serialization.

This is something I discussed in a previous post so there’s no point repeating it.

Cross-site Scripting (XSS)

Cross-site scripting, or XSS, is a complex attack. It involves injecting malicious scripts into websites that then run on every person’s browser visiting the page. This can lead to theft of user cookies, which in turn allows the attacker to impersonate users on the website. Protecting against XSS involves validating user-supplied data, treating it as display content, not executable code.

Let’s say I have a submit form that accepts user input that is saved to the database. Like the comments section in the blog. I can post in JavaScript code that would submit the user’s cookies to a site I control. Then I can steal this information and impersonate a user. This is a very common and surprising attack, it’s often performed by encoding the script into a link sent by email.

These are three types of XSS attacks:

• Stored XSS (Persistent) – The attack I described here is a form of stored XSS since the comment I would submit is saved in the database. At this point, every user that looks at the comment is attacked.

• Reflected XSS (Non-persistent) – In this form, the attacker sends a link to a user (or convinces the user to click on a link) that contains the malicious script. When the user clicks the link, the script runs, sending their data to the attacker. The script is embedded in the URL and reflected off the web server. This is usually part of a phishing attack.

• DOM-Based XSS – This type of attack occurs entirely in the victim’s browser. The web application’s client-side scripts write user-provided data to the Document Object Model. The data is subsequently read from the DOM by the web application and outputted to the browser. If the data was interpreted as JavaScript, it’s executed.

Protecting from XSS requires diligent validation of all input. We can protect against these attacks by checking if user-provided data is of the correct form and contains no malicious content. We must ensure any user-supplied content is treated as display content, not executable code.

There are many ways to validate user-submitted data and the Jsoup library contains one such API. Notice that Spring Boot contains XSS protection as part of the security configuration but I’ll cover that later.

personName = Jsoup.clean(personName, Whitelist.basic());

Notice that validating input is a recurring theme when it comes to security vulnerabilities. As developers we often strive to provide the most generic and flexible tooling, this works against us when it comes to security vulnerabilities. It’s important to limit input options even when we don’t see a problem.

Content-Security-Policy (CSP)

One of the ways to carry out an XSS attack is by including foreign code into our own website. One way to block this is using special HTTP headers to define which sites can include our site. This is a rather elaborate process but the nice thing is that Spring Security handles that nicely for us as well.

HttpOnly Cookies

Cookies can be created in the browser using JavaScript. This is a bad practice. Ideally, cookies should always come from the server and be marked as HTTP only (and HTTPS only). This blocks JavaScript code from accessing the cookie.

That means that even if a script is added somehow or a bad link is clicked, it won’t have access to the cookie value. This mitigates XSS attacks so even if your site vulnerable the attack can’t steal the cookie. We can enable HttpOnly cookies when we set the cookie in the server code.

Unvalidated Redirects and Forwards

Another security concern is unvalidated redirects and forwards. Here, an attacker creates a URL that looks like it’s coming from your domain, but redirects to another malicious site. The solution lies in validating and restricting included or submitted URLs, and never sending users blindly to third-party sites.

Lets say we have a login page. After we login we’re shown a splash screen and then we’re sent to the actual destination. This seems simple enough but some people need to go to page X and others need to go to page Y. We want to keep the code generic so we accept the destination URL as an argument. That way the login code can decide where to go next and we don’t need to know about all the user types e.g.: https://bug.com/postLogin?dest=url.

The problem is that a person can create a URL that looks like it’s coming from our domain, but pass in another URL as the last argument. Our users can end up on a malicious site without realizing they were redirected to a new site.

The solution is to validate and restrict included or submitted URLs and never send a user blindly to a third-party site.

Server Side Request Forgery (SSRF)

SSRF attacks are similar conceptually, in these attacks our server performs a request based on the request we received. Our server can be manipulated to request arbitrary URLs for an attacker. This can serve as the basis for information theft, denial of service attacks, etc.

Cross-Site Request Forgery (CSRF)

CSRF is another challenging issue where an attacker tricks users into hacking their own account. Typically, we’re logged into a website. Our credentials and cookies are already set. If a different website knows we’re logged in to a website it can trick us and get us to hack ourselves…

Let’s say you visit a website and it has a big button that you can press for your chance to win a million dollars. Would you press it?

What’s the harm right?

If that button is a form that submits the request directly to your bank, this can be used to steal currency and more.

The standard solution is to add a server-generated token into the HTML that changes with every request, thus validating that the HTML came from the legitimate site. This is a standard strategy supported by Spring Security.

We can also set our cookies to the SameSite policy which will mean a user won’t be logged in if he’s on a separate site. Turning this on for your login information is probably a good idea.

Final Word

In conclusion, while we did not delve into a lot of code writing in this post, the objective was to shed light on common security vulnerabilities and attacks, and how to prevent them. Understanding these concepts is fundamental in building secure applications, and the more we’re aware, the better equipped we are to thwart potential threats.

There are many tools for security validation, if you use a decent linter like SonarQube you would be on your way to a more secure app. Snyk also has great tooling that can help catch various vulnerabilities.

This paragraph from the post probably sums up the most important aspects:

Notice that validating input is a recurring theme when it comes to security vulnerabilities. As developers we often strive to provide the most generic and flexible tooling, this works against us when it comes to security vulnerabilities. It’s important to limit input options even when we don’t see a problem.

The post Understanding Security Vulnerabilities: A First Step in Preventing Attacks appeared first on ProdSens.live.