This talk explores big data using vector functional programming. These languages enable rapid interactive development of concise and efficient applications. Vectors are a natural low impedance abstraction for dealing with high performance relational and NoSQL column stores. We use the k language and kdb+ database to illustrate the vector applicative style and idioms of FP through examples working efficiently with big data.

Vector languages are heavily used in finance and insurance for analysis of large time series. Vector/Array dynamic languages originated with APL family and include NIAL, J and k and databases such as Kdb+, MonetDB and Vectorwise. The have inspired array extensions in other general languages, such as Python NumPy and NamedTuple; and special purpose GPUs. Often these languages are considered high barrier languages because the compact notation and vector thinking require an investment to develop proficiently. Broaden your FP knowledge and understand the joy and power vector FP.

Dave Thomas: NewSQL - The New Universal Query Language for Everything.

Back to the Future! In 2020 many new and legacy data systems, such as Spark, Teradata, Mongo, Cassandra, Splunk, Kafka, Flink, Azure, Google, Oracle…, will support a NewSQL.

We begin with a very brief review of the current state of data engineering practice - NoSQL, Big Data, Streaming, Time Series, BASE versus ACID, Graphs; DSLs, Map Reduce, Functional Programming etc. We look at the challenges presented by the plethora of different data formats, languages/tools and eventual consistency.

What if we didn’t have to deal with the challenges of sharding and eventual consistency? We discuss the recent increased use of new distributed ACID databases such as Google Spanner, AWS AuroraDB, Azure CosmosDB etc.

What if we didn’t need to worry about different languages, APIs for programming different databases? What if we could use one language for both batch and streaming? Very recently, at the ACM SIGMOD 2019 database conference, a new emerging data language was presented which seeks to provide a solution.

We refer to this emerging standard as NewSQL, which extends SQL to deal with both streaming and batch. It removes major limitations in SQL and provides the additional capabilities to SQL needed to deal with the velocity, volume and variety of diverse data sources. In this talk, we describe the evolution and major features of the new language. We conclude with a brief discussion of the impact on data engineering, data science and data consumers.

Danial Tham: Mystique - The Fight Against Rego Plate Cloning

In the rise of AI technologies, there is no doubt that privacy is a very popular topic which normally revolves around the negative impact of AI to our privacy. However despite the norm, we at Carsales built and deployed an AI tech called Mystique which fights crimes and protects user privacy. Mystique detects a rego plate in a photo and blurs it, protecting our private seller and dealer from rego plate cloning.

Rego plate cloning “involves falsifying a genuine number plate and attaching it to another vehicle which is often the same make, model and colour. The criminal may then use the car with cloned plates to drive on tollways without consequences, drive dangerously or conduct other criminal activity, with the genuine vehicle owner often sought to pay fines or tolls.”

In this talk, we are going to share the story about how we built and delivered Mystique and more importantly what lessons we learned in deploying tech at this scale, which processes 150,000 photos a day and touches many areas of business at Carsales.

Dave Thomas: NewSQL - The New Universal Query Language for Everything.

Back to the Future! In 2020 many new and legacy data systems, such as Spark, Teradata, Mongo, Cassandra, Splunk, Kafka, Flink, Azure, Google, Oracle…, will support a NewSQL.

We begin with a very brief review of the current state of data engineering practice - NoSQL, Big Data, Streaming, Time Series, BASE versus ACID, Graphs; DSLs, Map Reduce, Functional Programming etc. We look at the challenges presented by the plethora of different data formats, languages/tools and eventual consistency.

What if we didn’t have to deal with the challenges of sharding and eventual consistency? We discuss the recent increased use of new distributed ACID databases such as Google Spanner, AWS AuroraDB, Azure CosmosDB etc.

What if we didn’t need to worry about different languages, APIs for programming different databases? What if we could use one language for both batch and streaming? Very recently, at the ACM SIGMOD 2019 database conference, a new emerging data language was presented which seeks to provide a solution.

We refer to this emerging standard as NewSQL, which extends SQL to deal with both streaming and batch. It removes major limitations in SQL and provides the additional capabilities to SQL needed to deal with the velocity, volume and variety of diverse data sources. In this talk, we describe the evolution and major features of the new language. We conclude with a brief discussion of the impact on data engineering, data science and data consumers.

Agustinus Nalwan: Mystique - The Fight Against Rego Plate Cloning

In the rise of AI technologies, there is no doubt that privacy is a very popular topic which normally revolves around the negative impact of AI to our privacy. However despite the norm, we at Carsales built and deployed an AI tech called Mystique which fights crimes and protects user privacy. Mystique detects a rego plate in a photo and blurs it, protecting our private seller and dealer from rego plate cloning.

Rego plate cloning “involves falsifying a genuine number plate and attaching it to another vehicle which is often the same make, model and colour. The criminal may then use the car with cloned plates to drive on tollways without consequences, drive dangerously or conduct other criminal activity, with the genuine vehicle owner often sought to pay fines or tolls.”

In this talk, we are going to share the story about how we built and delivered Mystique and more importantly what lessons we learned in deploying tech at this scale, which processes 150,000 photos a day and touches many areas of business at Carsales.

Dave Thomas: NewSQL - The New Universal Query Language for Everything.

Back to the Future! In 2020 many new and legacy data systems, such as Spark, Teradata, Mongo, Cassandra, Splunk, Kafka, Flink, Azure, Google, Oracle…, will support a NewSQL.

We begin with a very brief review of the current state of data engineering practice - NoSQL, Big Data, Streaming, Time Series, BASE versus ACID, Graphs; DSLs, Map Reduce, Functional Programming etc. We look at the challenges presented by the plethora of different data formats, languages/tools and eventual consistency.

What if we didn’t have to deal with the challenges of sharding and eventual consistency? We discuss the recent increased use of new distributed ACID databases such as Google Spanner, AWS AuroraDB, Azure CosmosDB etc.

What if we didn’t need to worry about different languages, APIs for programming different databases? What if we could use one language for both batch and streaming? Very recently, at the ACM SIGMOD 2019 database conference, a new emerging data language was presented which seeks to provide a solution.

We refer to this emerging standard as NewSQL, which extends SQL to deal with both streaming and batch. It removes major limitations in SQL and provides the additional capabilities to SQL needed to deal with the velocity, volume and variety of diverse data sources. In this talk, we describe the evolution and major features of the new language. We conclude with a brief discussion of the impact on data engineering, data science and data consumers.

Agustinus Nalwan: Mystique - The Fight Against Rego Plate Cloning

In the rise of AI technologies, there is no doubt that privacy is a very popular topic which normally revolves around the negative impact of AI to our privacy. However despite the norm, we at Carsales built and deployed an AI tech called Mystique which fights crimes and protects user privacy. Mystique detects a rego plate in a photo and blurs it, protecting our private seller and dealer from rego plate cloning.

Rego plate cloning “involves falsifying a genuine number plate and attaching it to another vehicle which is often the same make, model and colour. The criminal may then use the car with cloned plates to drive on tollways without consequences, drive dangerously or conduct other criminal activity, with the genuine vehicle owner often sought to pay fines or tolls.”

In this talk, we are going to share the story about how we built and delivered Mystique and more importantly what lessons we learned in deploying tech at this scale, which processes 150,000 photos a day and touches many areas of business at Carsales.

NoSQL is slaying the Relational Database (RDB) dragons with their fast columns, simple key values, graphs, BASE, and schema less features throwing their hooked on ACID SQL row farmers and DBAs into disarray. But Wait! Rumours are emerging of a secret YesSQL tech heading at warp speed to the lucrative Copious Data Galaxy. Is NewSQL just RDB vendor FUD or does the Empire have any hot cool stuff? Attendees may choose be seated in columns or rows according to their preference!

In this talk we look at the exciting evolution of databases taking place to deal the challenge of copious data. Attendees will learn about the innovations of new database technology and the opportunities and challenges for using it in production and analytics applications.

NoSQL is slaying the Relational Database (RDB) dragons with their fast columns, simple key values, graphs, BASE, and schema less features throwing their hooked on ACID SQL row farmers and DBAs into disarray. But Wait! Rumours are emerging of a secret YesSQL tech heading at warp speed to the lucrative Copious Data Galaxy. Is NewSQL just RDB vendor FUD or does the Empire have any hot cool stuff? Attendees may choose be seated in columns or rows according to their preference!

In this talk we look at the exciting evolution of databases taking place to deal the challenge of copious data. Attendees will learn about the innovations of new database technology and the opportunities and challenges for using it in production and analytics applications.

NoSQL is slaying the Relational Database (RDB) dragons with their fast columns, simple key values, graphs, BASE, and schema less features throwing their hooked on ACID SQL row farmers and DBAs into disarray. But Wait! Rumours are emerging of a secret YesSQL tech heading at warp speed to the lucrative Copious Data Galaxy. Is NewSQL just RDB vendor FUD or does the Empire have any hot cool stuff? Attendees may choose be seated in columns or rows according to their preference!

In this talk we look at the exciting evolution of databases taking place to deal the challenge of copious data. Attendees will learn about the innovations of new database technology and the opportunities and challenges for using it in production and analytics applications.

It appears that there is a complexity conspiracy seeking to make the development of even straightforward applications difficult.

To be successful today one needs to cope with the full-stack - a constantly changing collection of languages, APIs, tools and frameworks.

It appears that complexity begets complexity? Yet most applications are not intrinsically complicated!

In this talk, we look at simple proven techniques which provide improved productivity and quality. They have stood the test of time, but unfortunately, are not well known to most developers. These techniques reduce the gap between the product owner and developer usually reducing amount code which must be developed. They have been applied to problems in almost every industry banking to real-time control. They don't depend on a particular language or technology hence large portions of an application can be moved easily from legacy technology to modern technology. We discuss the techniques and provide case studies of their use in major applications.

It appears that there is a complexity conspiracy seeking to make the development of even straightforward applications difficult.

To be successful today one needs to cope with the full-stack - a constantly changing collection of languages, APIs, tools and frameworks.

It appears that complexity begets complexity? Yet most applications are not intrinsically complicated!

In this talk, we look at simple proven techniques which provide improved productivity and quality. They have stood the test of time, but unfortunately, are not well known to most developers. These techniques reduce the gap between the product owner and developer usually reducing amount code which must be developed. They have been applied to problems in almost every industry banking to real-time control. They don't depend on a particular language or technology hence large portions of an application can be moved easily from legacy technology to modern technology. We discuss the techniques and provide case studies of their use in major applications.

Coming Soon...

And Now For Something Completely Different?

What do stock ticker feeds, web click streams, micropayments, online games, smart meters, and sensors on a formula one race car have in common? They all need to compute over and visualize large volumes of time series data. In this talk how one can interactively explore and visualize massive data sets using a vector functional programming.

Vector languages are heavily used in finance for the analysis of stock market data. They have inspired array capabilities in other languages, such as R, MatLab, Python NumPy, and Pandas. Vector languages and runtimes enable rapid interactive development of concise and efficient applications. Vectors are a natural low impedance abstraction for dealing with high-performance column stores. We introduce the key concepts of vector programming, applicative style, and idioms of FP. We then illustrate their use on typical data science problems exploring large data sets using visual analytics.

And Now For Something Completely Different?

What do stock ticker feeds, web click streams, micropayments, online games, smart meters, and sensors on a formula one race car have in common? They all need to compute over and visualize large volumes of time series data. In this talk how one can interactively explore and visualize massive data sets using a vector functional programming.

Vector languages are heavily used in finance for the analysis of stock market data. They have inspired array capabilities in other languages, such as R, MatLab, Python NumPy, and Pandas. Vector languages and runtimes enable rapid interactive development of concise and efficient applications. Vectors are a natural low impedance abstraction for dealing with high-performance column stores. We introduce the key concepts of vector programming, applicative style, and idioms of FP. We then illustrate their use on typical data science problems exploring large data sets using visual analytics.

And Now For Something Completely Different?

What do stock ticker feeds, web click streams, micropayments, online games, smart meters, and sensors on a formula one race car have in common? They all need to compute over and visualize large volumes of time series data. In this talk how one can interactively explore and visualize massive data sets using a vector functional programming.

Vector languages are heavily used in finance for the analysis of stock market data. They have inspired array capabilities in other languages, such as R, MatLab, Python NumPy, and Pandas. Vector languages and runtimes enable rapid interactive development of concise and efficient applications. Vectors are a natural low impedance abstraction for dealing with high-performance column stores. We introduce the key concepts of vector programming, applicative style, and idioms of FP. We then illustrate their use on typical data science problems exploring large data sets using visual analytics.

For the last decade, there has been increased concern about the integrity of capital markets. The crash of 2008-2009 and follow legal actions and press have created an image of a world of high-frequency traders who can leverage their computer power to manipulate markets. Technical talks on performance which is critical in finance, further characterize finance as hooked on speed/low latency. One gets the impression that fast data leads to a fast buck at public expenses. However, fast big data also enables the good guys!

We discuss how fast big data is being used in the financial industry to ensure good governance and protect consumers and businesses who depend on the integrity of financial markets. We discuss the better decisions enabled by algorithms; improved testing practices for algorithms; oversight of markets through surveillance; protection against cyber threats; and the use of data forensics to tell the true story of transactions past.

For the last decade, there has been increased concern about the integrity of capital markets. The crash of 2008-2009 and follow legal actions and press have created an image of a world of high-frequency traders who can leverage their computer power to manipulate markets. Technical talks on performance which is critical in finance, further characterize finance as hooked on speed/low latency. One gets the impression that fast data leads to a fast buck at public expenses. However, fast big data also enables the good guys!

We discuss how fast big data is being used in the financial industry to ensure good governance and protect consumers and businesses who depend on the integrity of financial markets. We discuss the better decisions enabled by algorithms; improved testing practices for algorithms; oversight of markets through surveillance; protection against cyber threats; and the use of data forensics to tell the true story of transactions past.

This is a workshop where we will seek to respond to challenges of attendees with respect to Scaling Agile development.  We begin with a quick discussion of transitioning to Lean and Agile in the Large to provide a common context. We then open the discussion for participants to bring forward their challenges. We vote on the challenges and then discuss them starting with the ones with the highest votes.  We time box the discussion of each challenge so we can deal with as many as possible.  Examples which may be discussed.

Transition Planning; Agile Maturity; Agile in a Legacy; Agile with Enterprise Vendors; Estimation in the Larger; Envisioning; Architecture and Design; Testing; Nonfunctional Requirements; Feature Teams versus Component Teams; Agile Database/BI; Portfolio Planning; Management and Technical Ladders; Metrics; Automation of Testing and Deployment....

Bring your challenges.

Agile, OOP... are like good hygiene in the kitchen, it results in meals with consistent quality and predictable prep and service times. It doesn't result in great meals nor substantially impact the ROI! Lean Thinking clearly shows that the only way to make a significant impact is to improve the value chain by improving flow. If everyone is following best practices no one has competitive advantage. Major improvements in the value chain depend on continued disruptive innovations. Innovations leverage people and their ideas. We use case studies to illustrate the different business and technical innovations and their impact. We conclude with a discussion of how to build and leverage an innovation culture versus a sprint death march when dealing with high value time to market projects.

Agile, OOP... are like good hygiene in the kitchen, it results in meals with consistent quality and predictable prep and service times. It doesn't result in great meals nor substantially impact the ROI! Lean Thinking clearly shows that the only way to make a significant impact is to improve the value chain by improving flow. If everyone is following best practices no one has competitive advantage. Major improvements in the value chain depend on continued disruptive innovations. Innovations leverage people and their ideas. We use case studies to illustrate the different business and technical innovations and their impact. We conclude with a discussion of how to build and leverage an innovation culture versus a sprint death march when dealing with high value time to market projects.