The Programmer News Hubb
Advertisement Banner
  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact
No Result
View All Result
  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact
No Result
View All Result
Gourmet News Hubb
No Result
View All Result
Home Technical Insights

SD Times Open-Source Project of the Week: AvroTensorDataset

admin by admin
June 18, 2023
in Technical Insights


Earlier this week, LinkedIn announced it was open-sourcing AvroTensorDataset, which is a “TensorFlow dataset for reading, parsing, and processing Avro data.” Apache Avro is the primary storage format that LinkedIn uses for its training data. 

According to LinkedIn, it was experiencing bottlenecks in its machine learning workloads that were caused by the need to read multiple terabytes of input data. AvroTensorDataset can speed up preprocessing of data by multiple orders of magnitude, according to the company.

The tool was built internally at LinkedIn, and it wanted to open-source the project so that others could experience the large performance boosts to training workloads. It has been in production for over a year already at LinkedIn. 

LinkedIn says that with this tool it has been able to improve processing speed by 162x compared to existing solutions and has decreased overall training time by 66%

“ATDSDataset is LinkedIn’s solution to efficiently read Avro data into TensorFlow. Through multiple performance enhancements, we were able to speed up I/O throughput by orders of magnitude over existing Avro reader solutions. Our team at LinkedIn worked closely with the TensorFlow I/O community to open-source this feature, and we hope that by open-sourcing it, the TensorFlow community can also benefit from these performance enhancements,” Jonathan Hung, staff software engineer at LinkedIn, wrote in a blog post. 



Source link

Previous Post

EdX launches new learning center with career resources

Next Post

Economic turmoil drives technical debt: Here’s how to manage it

Next Post

Economic turmoil drives technical debt: Here’s how to manage it

Recommended

An Introduction, and Alternatives — SitePoint

6 months ago

Weekly News for Designers № 672

10 months ago

20 Best Free Lightroom Mobile Presets for Stunning Photo Effects

4 days ago

The Top Five WordPress Contact Form Plugins — SitePoint

6 months ago

WordPress Developer Blog | CSS-Tricks

10 months ago

Time to hide your API

8 months ago

© The Programmer News Hubb All rights reserved.

Use of these names, logos, and brands does not imply endorsement unless specified. By using this site, you agree to the Privacy Policy and Terms & Conditions.

Navigate Site

  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact

Newsletter Sign Up.

No Result
View All Result
  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact

© 2022 The Programmer News Hubb All rights reserved.