Homepage of Prof. Dr. Thomas Kuhr

Homepage of Prof. Dr. Thomas Kuhr

Prof. Dr. Thomas Kuhr
Ludwig Maximilians University Munich
Fakultät für Physik
Excellence Cluster Universe
Boltzmannstr. 2
85748 Garching
Germany

Phone: +49 89 35831 7174

E-mail: Thomas.Kuhr@lmu.de

Experiments:

Research

Teaching

Talks

Publications

Proceedings

Bachelor and master thesis topics on data analysis and the development of analysis algorithms are offered on the web page of the

Experimental Flavor Physics Group

Research Projects

Search for new physics in (rare) decays

Belle: B -> K(*) nu nu, B -> tau nu, B -> D(*) tau nu (Nature Article, Scientific American article)
Belle: Improved full reconstruction
Search for new physics with time dependent CP violation measurements

Belle: B -> D(*)D, B -> D* D*, B -> phi K pi
CDF: Bs -> J/psi phi (Fermilab Today: Result of the Week), Bs lifetime in Bs -> J/psi f0
Belle/Belle II and CDF: Improved flavor tagging with neural networks
Measurement of Bs mixing

CDF: Observation of Bs oscillations (Fermilab Press Release)
Heavy hadron spectroscopy

CDF: Evidence for a new B pi resonance
Belle: Search for Y(4140) -> J/psi phi
CDF: Study of X(3872) particle: mass (Fermilab Today: Result of the Week), quantum numbers
CDF: Study of orbitally excited B mesons: B**0, B**+, Bs**0
CDF: Study of excited charm baryons: &Lambdac(2595), &Lambdac(2625), &Sigmac(2455) and &Sigmac(2520)
Preparations for future research projects

Belle II: Reconstruction and analysis algorithms
Belle II: Computing infrastructure and software development
Teaching

Lectures

E5/E5p: Kern- und Teilchenphysik (LSF) (WS 17/18, WS 16/17)
Heavy Quark Physics (LSF) (SS 17)
Physik der schweren Quarks (LSF) (SS 16, WS 15/16)
Seminars

Big Data Science in und außerhalb der Physik (WS 17/18)
Key Experiments in Particle Physics (SS 17)
Flavor Lunch (each semester)
Belle Paper Seminar (each semester)
Blockseminar: Methoden der experimentellen Teilchenphysik am Beispiel von Belle (II) (SS 15)
Recent Talks

Belle II early physics program
Blois 2017, Blois, 31 May 2017
From hadron colliders to e+e-, flavour physics at Belle II
LHCb Workshop, Neckarzimmern, 23 Mar 2017
The Belle II Software – From Detector Signals to Physics Results
INSTR17, Novosibirsk, 27 Feb 2017
Overview Research Area C
Universe Cluster Science Week, Garching, 8 Dec 2016
New Physics Searches at Belle II
Research Area B Science Day, Garching, 13 Oct 2016
Event Visualization in Belle II and Genfit
Interdisciplinary Cluster Workshop on Visualization, Garching, 4 Nov 2015
New B -> D(*) tau nu results from Belle and LHCb
MIAPP: Indirect Searches for New Physics in the LHC and Flavour Precision Era, Garching, 8 Jun 2015
The Next Generation B-Factory Experiment Belle II
MLL Kolloquium, Garching, 7 May 2015
New B -> D(*) tau nu result from Belle
FPCP2015, Nagoya, 25 May 2015
B -> D(*) tau nu and B -> tau nu at Belle
Tau 2014, Aachen, 14 Sep 2014

Computing at Belle II
Terascale Alliance Workshop, Karlsruhe, 3 Dec 2013

Belle II: The Future of Flavor Physics
Antrittsvorlesung, Karlsruhe, 6 Nov 2013

First Production with the Belle II Distributed Computing System
CHEP 2013, Amsterdam, 14 Oct 2013

B -> h(*) nu nu-bar
2nd KEK Flavor Factory Workshop, Tsukuba, 13 Mar 2013

The Next Generation B Factory Experiment Belle II
Prisma + GK-Seminar, Mainz, 5 Dec 2012

Belle II Status and Plans
Capri Workshop, Anacapri, 11 Jun 2012

Computing at Belle II
CHEP 2012, New York, 22 May 2012

Updated Search for B->mu+mu- Decays at CDF
EPS 2011, Grenoble, 22 Jul 2011

Heavy-flavor Baryons
Hadron 2011, Munich, 16 Jun 2011

SuperKEKB and Belle II
LAPP Experimental Physics Seminar, Annecy, 10 Dec 2010

Computing at Belle II
CHEP2010, Taipei, 18 Oct 2010

Recent Experimental B Physics Results
EuroFlavour2010, München, 8 Sep 2010

Bs Physics
PIC2010, Karlsruhe, 2 Sep 2010

Computing at Belle II
DESY Computing Seminar, Hamburg, 14 Jun 2010

Status of SuperKEKB and Belle II
FPCP2010, Torino, 29 May 2010

Upsilon Polarization Measurement at CDF
DIS, Florence, 22 Apr 2010

Status and Prospects of Belle and Belle II
Seminar, Wuppertal, 10 Dec 2009

Search for New Physics in the Bs Sector at the Tevatron and
New Charmonium-like States at B Factories
DIS, Madrid, 29 and 27 Apr 2009

Hints for New Physics in Bs Mixing?
Hints for New Physics in Flavor Decays, KEK, 20 Mar 2009
Heavy-Flavour-Physik bei CDF (Eingeladener Vortrag)
DPG Tagung, München, 10 Mar 2009

Quarkonium Spectroscopy and Decay Results from CDF
Quarkonium Workshop, Nara, 3 Dec 2008

Flavor-Physik bei CDF
Physikalisches Kolloquium, Karlsruhe, 7 Nov 2008

Rare B and Charm Decays
Moriond QCD, La Thuile, 9 Mar 2008

B Baryon Spectroscopy
Lattice QCD Workshop, FNAL, 11 Dec 2007

Lifetime Difference and CP Asymmetry in the B0s -> J/psi phi decay
SUSY, Karlsruhe, 31 Jul 2007

Studies of B States at the Tevatron
La Thuile, 9 Mar 2007

Publications

Flavor Physics at the Tevatron
Springer Tracts in Modern Physics (2012)

A challenge to lepton universality in B-meson decays
Nature 546, 227 (2017)

Averages of b-hadron, c-hadron, and tau-lepton properties as of summer 2016
Accepted by European Physical Journal C (2017)

Measurement of the branching ratio of Bbar -> D(*) tau- nubar_tau relative to Bbar -> D(*) l- nubar_l decays with hadronic tagging at Belle
Phys. Rev. D. 92, 072014 (2015)

Measurement of the branching fraction of B+ -> tau+ nu_tau decays with the semileptonic tagging method
Phys. Rev. D 92, 051102(R) (2015) PRD Editors Suggestion.

Search for B+ -> l+ nu gamma decays with hadronic tagging using the full Belle data sample
Phys. Rev. D 91, 112009 (2015).

Study of orbitally excited B mesons and evidence for a new B pi resonance
Phys. Rev. D 90, 012013 (2014).

Angular analysis of B0 -> phi K* decays and search for CP violation at Belle
Phys. Rev. D 88, 072004 (2013).

Search for B -> h(*) nu nubar with the full Belle data sample
Phys. Rev. D 87, 111103 (2013).

Measurement of CP-Violation Asymmetries in D0 -> K_S pi+ pi-
Phys. Rev. D 86, 032007 (2012)

First Observation of CP Violation and Improved Measurement of the Branching Fraction and Polarization of B0 -> D*+ D*- Decays
Phys. Rev. D 86, 071103 (2012)

Measurements of Fractions and Time-dependent CP Violating Asymmetries in B0 -> D(*)+- D-+
Phys. Rev. D 85, 091106 (2012)

Measurement of Bs0 -> Ds(*)+ Ds(*)- Branching Ratios
Phys. Rev. Lett. 108, 201801 (2012)

Measurement of the CP-Violating Phase beta_s in B0s -> J/psi phi Decays with the CDF II Detector
Phys. Rev. D 85, 072002 (2012)

Measurement of branching ratio and Bs0 lifetime in the decay Bs0 -> J/psi f0(980) at CDF
Phys. Rev. D 84, 052012 (2011)

A Hierarchical NeuroBayes-based Algorithm for Full Reconstruction of B Mesons at B Factories
Nucl. Instrum. Meth. A 654:432-440 (2011)

Measurements of the properties of Lambda_c(2595), Lambda_c(2625), Sigma_c(2455), and Sigma_c(2520) baryons
Phys. Rev. D 84, 012003 (2011)

Belle II Technical Design Report
KEK Report 2010-1 (2010)

Formulae for the Analysis of the Flavor-Tagged Decay B0s -> J/psi phi
JHEP 1011:158 (2010)

Precision Measurement of the X(3872) Mass in J/psi pi+ pi- Decays
Phys. Rev. Lett. 103, 152001 (2009)

Measurement of Resonance Parameters of Orbitally Excited Narrow B0 Mesons
Phys. Rev. Lett. 102, 102003 (2009)

Measurement of Lifetime and Decay-Width Difference in B0s -> J/psi phi Decays
Phys. Rev. Lett. 100, 121803 (2008)

Observation of Orbitally Excited Bs Mesons
Phys. Rev. Lett. 100, 082001 (2008)

Analysis of the Quantum Numbers JPC of the X(3872)
Phys. Rev. Lett. 98, 132002 (2007)

Observation of Bs Bsbar Oscillations
Phys. Rev. Lett. 97, 242003 (2006)

ALICE Physics Performance Report, Volume I
J. Phys. G30, 1517 (2004)

Proceedings

The Belle II Software – From Detector Signals to Physics Result
INSTR 2017, Novosibirsk, 27 Feb – 3 Mar 2017

Belle II Software
ACAT 2016, Valparaiso 18-22 Jan 2016

New B -> D(*) tau nu result from Belle
FPCP 2015, Nagoya, 25-29 May 2015

Software Development at Belle II
CHEP 2015, Okinawa, 13-17 Apr 2015

First production with the Belle II distributed computing system
CHEP 2013, Amsterdam, 14-18 Oct 2013

Belle II Status and Plans
Capri 2012, Anacapri, 11-13 Jun 2012

Computing at Belle II
CHEP 2012, New York, 21-25 May 2012

Updated Search for B(s)->mu+mu- Decays at CDF
EPS, Grenoble, 21-27 Jul, 2011

Heavy Flavor Baryons at the Tevatron
Hadron, Munich, 13-17 Jun, 2011

Computing at Belle II
CHEP2010, Taipei, 18-22 Oct 2010

Bs Physics
PIC, Karlsruhe, 1-4 Sep 2010

Status of SuperKEKB and Belle II
FPCP, Torino, 25-29 May 2010

Upsilon Polarization Measurement at CDF
DIS, Florence, 19-23 Apr 2010

Search for New Physics in the Bs Sector at the Tevatron
DIS, Madrid, 26-30 Apr 2009

New Charmonium-like States at B-Factories
DIS, Madrid, 26-30 Apr 2009

Rare B and Charm Decays at the Tevatron
Moriond QCD, La Thuile, 8-15 Mar 2008

Lifetime Difference and CP Asymmetry in the B0s -> J/psi phi decay
SUSY, Karlsruhe, 26 Jul – 1 Aug 2007

Studies of B States at the Tevatron
La Thuile, 4-10 Mar 2007

Bayesian approach for combined particle identification in ALICE experiment at LHC
CHEP, Interlaken, 27 Sep – 1 Oct 2004

Track reconstruction in high density environment
CHEP, Interlaken, 27 Sep – 1 Oct 2004

Open beauty photoproduction at HERA
DIS, Liverpool, 25-30 Apr 2000

Useful One-Line Scripts for Perl

Useful One-Line Scripts for Perl

Useful One-Line Scripts for Perl                    Dec 03 2013 | version 1.10
--------------------------------                    -----------   ------------

Compiled by Peteris Krumins (peter@catonmat.net, @pkrumins on Twitter)
http://www.catonmat.net -- good coders code, great reuse

Latest version of this file is always at:

    http://www.catonmat.net/download/perl1line.txt

This file is also available in other languages:

    Chinese: https://github.com/vinian/perl1line.txt

    Please email me peter@catonmat.net if you wish to translate it.

Perl One-Liners on Github:
 
    https://github.com/pkrumins/perl1line.txt

    You can send me pull requests over GitHub! I accept bug fixes,
    new one-liners, translations and everything else related.

I have also written "Perl One-Liners Explained" ebook that's based on
this file. It explains all the one-liners here. Get it at:

    http://www.catonmat.net/blog/perl-book/

No Starch Press has published "Perl One-Liners" as a real book too:

    http://nostarch.com/perloneliners

These one-liners work both on UNIX systems and Windows. Most likely your
UNIX system already has Perl. For Windows get the Strawberry Perl at:

    http://www.strawberryperl.com/

Table of contents:

    1. File Spacing
    2. Line Numbering
    3. Calculations
    4. String Creation and Array Creation
    5. Text Conversion and Substitution
    6. Selective Printing and Deleting of Certain Lines    
    7. Handy Regular Expressions
    8. Perl tricks


FILE SPACING 
------------

# Double space a file
perl -pe '$\="\n"'
perl -pe 'BEGIN { $\="\n" }'
perl -pe '$_ .= "\n"'
perl -pe 's/$/\n/'
perl -nE 'say'

# Double space a file, except the blank lines
perl -pe '$_ .= "\n" unless /^$/'
perl -pe '$_ .= "\n" if /\S/'

# Triple space a file
perl -pe '$\="\n\n"'
perl -pe '$_.="\n\n"'

# N-space a file
perl -pe '$_.="\n"x7'

# Add a blank line before every line
perl -pe 's//\n/'

# Remove all blank lines
perl -ne 'print unless /^$/'
perl -lne 'print if length'
perl -ne 'print if /\S/'

# Remove all consecutive blank lines, leaving just one
perl -00 -pe ''
perl -00pe0

# Compress/expand all blank lines into N consecutive ones
perl -00 -pe '$_.="\n"x4'

# Fold a file so that every set of 10 lines becomes one tab-separated line
perl -lpe '$\ = $. % 10 ? "\t" : "\n"'


LINE NUMBERING
--------------

# Number all lines in a file
perl -pe '$_ = "$. $_"'

# Number only non-empty lines in a file
perl -pe '$_ = ++$a." $_" if /./'

# Number and print only non-empty lines in a file (drop empty lines)
perl -ne 'print ++$a." $_" if /./'

# Number all lines but print line numbers only non-empty lines
perl -pe '$_ = "$. $_" if /./'

# Number only lines that match a pattern, print others unmodified
perl -pe '$_ = ++$a." $_" if /regex/'

# Number and print only lines that match a pattern
perl -ne 'print ++$a." $_" if /regex/'

# Number all lines, but print line numbers only for lines that match a pattern
perl -pe '$_ = "$. $_" if /regex/'

# Number all lines in a file using a custom format (emulate cat -n)
perl -ne 'printf "%-5d %s", $., $_'

# Print the total number of lines in a file (emulate wc -l)
perl -lne 'END { print $. }'
perl -le 'print $n=()='
perl -le 'print scalar(()=)'
perl -le 'print scalar(@foo=)'
perl -ne '}{print $.'
perl -nE '}{say $.'

# Print the number of non-empty lines in a file
perl -le 'print scalar(grep{/./})'
perl -le 'print ~~grep{/./}'
perl -le 'print~~grep/./,'
perl -E 'say~~grep/./,'

# Print the number of empty lines in a file
perl -lne '$a++ if /^$/; END {print $a+0}'
perl -le 'print scalar(grep{/^$/})'
perl -le 'print ~~grep{/^$/}'
perl -E 'say~~grep{/^$/}'

# Print the number of lines in a file that match a pattern (emulate grep -c)
perl -lne '$a++ if /regex/; END {print $a+0}'
perl -nE '$a++ if /regex/; END {say $a+0}'


CALCULATIONS
------------

# Check if a number is a prime
perl -lne '(1x$_) !~ /^1?$|^(11+?)\1+$/ && print "$_ is prime"'

# Print the sum of all the fields on a line
perl -MList::Util=sum -alne 'print sum @F'

# Print the sum of all the fields on all lines
perl -MList::Util=sum -alne 'push @S,@F; END { print sum @S }'
perl -MList::Util=sum -alne '$s += sum @F; END { print $s }'

# Shuffle all fields on a line
perl -MList::Util=shuffle -alne 'print "@{[shuffle @F]}"'
perl -MList::Util=shuffle -alne 'print join " ", shuffle @F'

# Find the minimum element on a line
perl -MList::Util=min -alne 'print min @F'

# Find the minimum element over all the lines
perl -MList::Util=min -alne '@M = (@M, @F); END { print min @M }'
perl -MList::Util=min -alne '$min = min @F; $rmin = $min unless defined $rmin && $min > $rmin; END { print $rmin }'

# Find the maximum element on a line
perl -MList::Util=max -alne 'print max @F'

# Find the maximum element over all the lines
perl -MList::Util=max -alne '@M = (@M, @F); END { print max @M }'

# Replace each field with its absolute value
perl -alne 'print "@{[map { abs } @F]}"'

# Find the total number of fields (words) on each line
perl -alne 'print scalar @F'

# Print the total number of fields (words) on each line followed by the line
perl -alne 'print scalar @F, " $_"'

# Find the total number of fields (words) on all lines
perl -alne '$t += @F; END { print $t}'

# Print the total number of fields that match a pattern
perl -alne 'map { /regex/ && $t++ } @F; END { print $t }'
perl -alne '$t += /regex/ for @F; END { print $t }'
perl -alne '$t += grep /regex/, @F; END { print $t }'

# Print the total number of lines that match a pattern
perl -lne '/regex/ && $t++; END { print $t }'

# Print the number PI to n decimal places
perl -Mbignum=bpi -le 'print bpi(n)'

# Print the number PI to 39 decimal places
perl -Mbignum=PI -le 'print PI'

# Print the number E to n decimal places
perl -Mbignum=bexp -le 'print bexp(1,n+1)'

# Print the number E to 39 decimal places
perl -Mbignum=e -le 'print e'

# Print UNIX time (seconds since Jan 1, 1970, 00:00:00 UTC)
perl -le 'print time'

# Print GMT (Greenwich Mean Time) and local computer time
perl -le 'print scalar gmtime'
perl -le 'print scalar localtime'

# Print local computer time in H:M:S format
perl -le 'print join ":", (localtime)[2,1,0]'

# Print yesterday's date
perl -MPOSIX -le '@now = localtime; $now[3] -= 1; print scalar localtime mktime @now'

# Print date 14 months, 9 days and 7 seconds ago
perl -MPOSIX -le '@now = localtime; $now[0] -= 7; $now[4] -= 14; $now[7] -= 9; print scalar localtime mktime @now'

# Prepend timestamps to stdout (GMT, localtime)
tail -f logfile | perl -ne 'print scalar gmtime," ",$_'
tail -f logfile | perl -ne 'print scalar localtime," ",$_'

# Calculate factorial of 5
perl -MMath::BigInt -le 'print Math::BigInt->new(5)->bfac()'
perl -le '$f = 1; $f *= $_ for 1..5; print $f'

# Calculate greatest common divisor (GCM)
perl -MMath::BigInt=bgcd -le 'print bgcd(@list_of_numbers)'

# Calculate GCM of numbers 20 and 35 using Euclid's algorithm
perl -le '$n = 20; $m = 35; ($m,$n) = ($n,$m%$n) while $n; print $m'

# Calculate least common multiple (LCM) of numbers 35, 20 and 8
perl -MMath::BigInt=blcm -le 'print blcm(35,20,8)'

# Calculate LCM of 20 and 35 using Euclid's formula: n*m/gcd(n,m)
perl -le '$a = $n = 20; $b = $m = 35; ($m,$n) = ($n,$m%$n) while $n; print $a*$b/$m'

# Generate 10 random numbers between 5 and 15 (excluding 15)
perl -le '$n=10; $min=5; $max=15; $, = " "; print map { int(rand($max-$min))+$min } 1..$n'

# Find and print all permutations of a list
perl -MAlgorithm::Permute -le '$l = [1,2,3,4,5]; $p = Algorithm::Permute->new($l); print @r while @r = $p->next'

# Generate the power set
perl -MList::PowerSet=powerset -le '@l = (1,2,3,4,5); for (@{powerset(@l)}) { print "@$_" }'

# Convert an IP address to unsigned integer
perl -le '$i=3; $u += ($_<>8*($_))&0xFF) } reverse 0..3'
perl -le '$ip = 2130706433; $, = "."; print map { (($ip>>8*($_))&0xFF) } reverse 0..3'


STRING CREATION AND ARRAY CREATION
----------------------------------

# Generate and print the alphabet
perl -le 'print a..z'
perl -le 'print ("a".."z")'
perl -le '$, = ","; print ("a".."z")'
perl -le 'print join ",", ("a".."z")'

# Generate and print all the strings from "a" to "zz"
perl -le 'print ("a".."zz")'
perl -le 'print "aa".."zz"'

# Create a hex lookup table
@hex = (0..9, "a".."f")

# Convert a decimal number to hex using @hex lookup table
perl -le '$num = 255; @hex = (0..9, "a".."f"); while ($num) { $s = $hex[($num%16)&15].$s; $num = int $num/16 } print $s'
perl -le '$hex = sprintf("%x", 255); print $hex'
perl -le '$num = "ff"; print hex $num'

# Generate a random 8 character password
perl -le 'print map { ("a".."z")[rand 26] } 1..8'
perl -le 'print map { ("a".."z", 0..9)[rand 36] } 1..8'

# Create a string of specific length
perl -le 'print "a"x50'

# Create a repeated list of elements
perl -le '@list = (1,2)x20; print "@list"'

# Create an array from a string
@months = split ' ', "Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec"
@months = qw/Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec/

# Create a string from an array
@stuff = ("hello", 0..9, "world"); $string = join '-', @stuff

# Find the numeric values for characters in the string
perl -le 'print join ", ", map { ord } split //, "hello world"'

# Convert a list of numeric ASCII values into a string
perl -le '@ascii = (99, 111, 100, 105, 110, 103); print pack("C*", @ascii)'
perl -le '@ascii = (99, 111, 100, 105, 110, 103); print map { chr } @ascii'

# Generate an array with odd numbers from 1 to 100
perl -le '@odd = grep {$_ % 2 == 1} 1..100; print "@odd"'
perl -le '@odd = grep { $_ & 1 } 1..100; print "@odd"'

# Generate an array with even numbers from 1 to 100
perl -le '@even = grep {$_ % 2 == 0} 1..100; print "@even"'

# Find the length of the string
perl -le 'print length "one-liners are great"'

# Find the number of elements in an array
perl -le '@array = ("a".."z"); print scalar @array'
perl -le '@array = ("a".."z"); print $#array + 1'


TEXT CONVERSION AND SUBSTITUTION
--------------------------------

# ROT13 a string
'y/A-Za-z/N-ZA-Mn-za-m/'

# ROT 13 a file
perl -lpe 'y/A-Za-z/N-ZA-Mn-za-m/' file

# Base64 encode a string
perl -MMIME::Base64 -e 'print encode_base64("string")'
perl -MMIME::Base64 -0777 -ne 'print encode_base64($_)' file

# Base64 decode a string
perl -MMIME::Base64 -le 'print decode_base64("base64string")'
perl -MMIME::Base64 -ne 'print decode_base64($_)' file

# URL-escape a string
perl -MURI::Escape -le 'print uri_escape($string)'

# URL-unescape a string
perl -MURI::Escape -le 'print uri_unescape($string)'

# HTML-encode a string
perl -MHTML::Entities -le 'print encode_entities($string)'

# HTML-decode a string
perl -MHTML::Entities -le 'print decode_entities($string)'

# Convert all text to uppercase
perl -nle 'print uc'
perl -ple '$_=uc'
perl -nle 'print "\U$_"'

# Convert all text to lowercase
perl -nle 'print lc'
perl -ple '$_=lc'
perl -nle 'print "\L$_"'

# Uppercase only the first word of each line
perl -nle 'print ucfirst lc'
perl -nle 'print "\u\L$_"'

# Invert the letter case
perl -ple 'y/A-Za-z/a-zA-Z/'

# Camel case each line
perl -ple 's/(\w+)/\u$1/g'
perl -ple 's/(?<!['])(\w+)/\u\1/g'

# Strip leading whitespace (spaces, tabs) from the beginning of each line
perl -ple 's/^[ \t]+//'
perl -ple 's/^\s+//'

# Strip trailing whitespace (space, tabs) from the end of each line
perl -ple 's/[ \t]+$//'

# Strip whitespace from the beginning and end of each line
perl -ple 's/^[ \t]+|[ \t]+$//g'

# Convert UNIX newlines to DOS/Windows newlines
perl -pe 's|\n|\r\n|'

# Convert DOS/Windows newlines to UNIX newlines
perl -pe 's|\r\n|\n|'

# Convert UNIX newlines to Mac newlines
perl -pe 's|\n|\r|'

# Substitute (find and replace) "foo" with "bar" on each line
perl -pe 's/foo/bar/'

# Substitute (find and replace) all "foo"s with "bar" on each line
perl -pe 's/foo/bar/g'

# Substitute (find and replace) "foo" with "bar" on lines that match "baz"
perl -pe '/baz/ && s/foo/bar/'

# Binary patch a file (find and replace a given array of bytes as hex numbers)
perl -pi -e 's/\x89\xD8\x48\x8B/\x90\x90\x48\x8B/g' file


SELECTIVE PRINTING AND DELETING OF CERTAIN LINES
------------------------------------------------

# Print the first line of a file (emulate head -1)
perl -ne 'print; exit'

# Print the first 10 lines of a file (emulate head -10)
perl -ne 'print if $. <= 10'
perl -ne '$. = 80'

# Print lines that are less than 80 chars in length
perl -ne 'print if length = 17 && $.  length($l); END { print $l }'

# Print the shortest line
perl -ne '$s = $_ if $. == 1; $s = $_ if length($_) < length($s); END { print $s }'

# Print all lines that contain a number
perl -ne 'print if /\d/'

# Find all lines that contain only a number
perl -ne 'print if /^\d+$/'

# Print all lines that contain only characters
perl -ne 'print if /^[[:alpha:]]+$/

# Print every second line
perl -ne 'print if $. % 2'

# Print every second line, starting the second line
perl -ne 'print if $. % 2 == 0'

# Print all lines that repeat
perl -ne 'print if ++$a{$_} == 2'

# Print all unique lines
perl -ne 'print unless $a{$_}++'

# Print the first field (word) of every line (emulate cut -f 1 -d ' ')
perl -alne 'print $F[0]'


HANDY REGULAR EXPRESSIONS
-------------------------

# Match something that looks like an IP address
/^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/
/^(\d{1,3}\.){3}\d{1,3}$/

# Test if a number is in range 0-255
/^([0-9]|[0-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$/

# Match an IP address
my $ip_part = qr|([0-9]|[0-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])|;
if ($ip =~ /^($ip_part\.){3}$ip_part$/) {
 say "valid ip";
}

# Check if the string looks like an email address
/\S+@\S+\.\S+/

# Check if the string is a decimal number
/^\d+$/
/^[+-]?\d+$/
/^[+-]?\d+\.?\d*$/

# Check if the string is a hexadecimal number
/^0x[0-9a-f]+$/i

# Check if the string is an octal number
/^0[0-7]+$/

# Check if the string is binary
/^[01]+$/

# Check if a word appears twice in the string
/(word).*\1/

# Increase all numbers by one in the string
$str =~ s/(\d+)/$1+1/ge

# Extract HTTP User-Agent string from the HTTP headers
/^User-Agent: (.+)$/

# Match printable ASCII characters
/[ -~]/

# Match unprintable ASCII characters
/[^ -~]/

# Match text between two HTML tags
m|<strong>([^&lt;]*)</strong>|
m|<strong>(.*?)</strong>|

# Replace all <b> tags with <strong>
$html =~ s|||g

# Extract all matches from a regular expression
my @matches = $text =~ /regex/g;


PERL TRICKS
-----------

# Print the version of a Perl module
perl -MModule -le 'print $Module::VERSION'
perl -MLWP::UserAgent -le 'print $LWP::UserAgent::VERSION'


PERL ONE-LINERS EXPLAINED E-BOOK
--------------------------------

I have written an ebook based on the one-liners in this file. If you wish to
support my work and learn more about these one-liners, you can get a copy
of my ebook at:

    http://www.catonmat.net/blog/perl-book/

The ebook is based on the 7-part article series that I wrote on my blog.
In the ebook I reviewed all the one-liners, improved explanations, added
new ones, and added two new chapters - introduction to Perl one-liners
and summary of commonly used special variables.

You can read the original article series here:

    http://www.catonmat.net/blog/perl-one-liners-explained-part-one/
    http://www.catonmat.net/blog/perl-one-liners-explained-part-two/
    http://www.catonmat.net/blog/perl-one-liners-explained-part-three/
    http://www.catonmat.net/blog/perl-one-liners-explained-part-four/
    http://www.catonmat.net/blog/perl-one-liners-explained-part-five/
    http://www.catonmat.net/blog/perl-one-liners-explained-part-six/
    http://www.catonmat.net/blog/perl-one-liners-explained-part-seven/


CREDITS
-------

Andy Lester       http://www.petdance.com
Shlomi Fish       http://www.shlomifish.org
Madars Virza      http://www.madars.org
caffecaldo        https://github.com/caffecaldo
Kirk Kimmel       https://github.com/kimmel
avar              https://github.com/avar
rent0n


FOUND A BUG? HAVE ANOTHER ONE-LINER?
------------------------------------

Email bugs and new one-liners to me at peter@catonmat.net!


HAVE FUN
--------

I hope you found these one-liners useful. Have fun!

#---end of file---

Position Independent Code (PIC) in shared libraries

Position Independent Code (PIC) in shared libraries

Toggle navigation
Eli Bendersky’s website
Position Independent Code (PIC) in shared libraries
November 03, 2011 at 06:14 Tags Articles , Assembly , C & C++ , Linkers and loaders , Linux
I’ve described the need for special handling of shared libraries while loading them into the process’s address space in a previous article. Briefly, when the linker creates a shared library, it doesn’t know in advance where it might be loaded. This creates a problem for the data and code references within the library, which should be somehow made to point to the correct memory locations.

There are two main approaches to solve this problem in Linux ELF shared libraries:

Load-time relocation
Position independent code (PIC)
Load-time relocation was already covered. Here, I want to explain the second approach – PIC.

I originally planned to focus on both x86 and x64 (a.k.a. x86-64) in this article, but as it grew longer and longer I decided it won’t be practical. So, it will explain only how PIC works on x86, picking this older architecture specifically because (unlike x64) it wasn’t designed with PIC in mind, so implementing PIC on it is a bit trickier. A future (hopefully much shorter) article will build upon the foundation of this one to explain how PIC is implemented on x64.

Some problems of load-time relocation
As we’ve seen in the previous article, load-time relocation is a fairly straightforward method, and it works. PIC, however, is much more popular nowadays, and is usually the recommended method of building shared libraries. Why is this so?

Load-time relocation has a couple of problems: it takes time to perform, and it makes the text section of the library non-shareable.

First, the performance problem. If a shared library was linked with load-time relocation entries, it will take some time to actually perform these relocations when the application is loaded. You may think that the cost shouldn’t be too large – after all, the loader doesn’t have to scan through the whole text section – it should only look at the relocation entries. But if a complex piece of software loads multiple large shared libraries at start-up, and each shared library must first have its load-time relocations applied, these costs can build up and result in a noticeable delay in the start-up time of the application.

Second, the non-shareable text section problem, which is somewhat more serious. One of the main points of having shared libraries in the first place, is saving RAM. Some common shared libraries are used by multiple applications. If the text section (where the code is) of the shared library can only be loaded into memory once (and then mapped into the virtual memories of many processes), considerable amounts of RAM can be saved. But this is not possible with load-time relocation, since when using this technique the text section has to be modified at load-time to apply the relocations. Therefore, for each application that loaded this shared library, it will have to be wholly placed in RAM again [1]. Different applications won’t be able to really share it.

Moreover, having a writable text section (it must be kept writable, to allow the dynamic loader to perform the relocations) poses a security risk, making it easier to exploit the application.

As we’ll see in this article, PIC mostly mitigates these problems.

PIC – introduction
The idea behind PIC is simple – add an additional level of indirection to all global data and function references in the code. By cleverly utilizing some artifacts of the linking and loading processes, it’s possible to make the text section of the shared library truly position independent, in the sense that it can be easily mapped into different memory addresses without needing to change one bit. In the next few sections I will explain in detail how this feat is achieved.

Key insight #1 – offset between text and data sections
One of the key insights on which PIC relies is the offset between the text and data sections, known to the linker at link-time. When the linker combines several object files together, it collects their sections (for example, all text sections get unified into a single large text section). Therefore, the linker knows both about the sizes of the sections and about their relative locations.

For example, the text section may be immediately followed by the data section, so the offset from any given instruction in the text section to the beginning of the data section is just the size of the text section minus the offset of the instruction from the beginning of the text section – and both these quantities are known to the linker.

In the diagram above, the code section was loaded into some address (unknown at link-time) 0xXXXX0000 (the X-es literally mean “don’t care”), and the data section right after it at offset 0xXXXXF000. Then, if some instruction at offset 0x80 in the code section wants to reference stuff in the data section, the linker knows the relative offset (0xEF80 in this case) and can encode it in the instruction.

Note that it wouldn’t matter if another section was placed between the code and data sections, or if the data section preceded the code section. Since the linker knows the sizes of all sections and decides where to place them, the insight holds.

Key insight #2 – making an IP-relative offset work on x86
The above is only useful if we can actually put the relative offset to work. But data references (i.e. in the mov instruction) on x86 require absolute addresses. So, what can we do?

If we have a relative address and need an absolute address, what’s missing is the value of the instruction pointer (since, by definition, the relative address is relative to the instruction’s location). There’s no instruction to obtain the value of the instruction pointer on x86, but we can use a simple trick to get it. Here’s some assembly pseudo-code that demonstrates it:

call TMPLABEL
TMPLABEL:
pop ebx
What happens here is:

The CPU executes call TMPLABEL, which causes it to save the address of the next instruction (the pop ebx) on stack and jump to the label.
Since the instruction at the label is pop ebx, it gets executed next. It pops a value from the stack into ebx. But this value is the address of the instruction itself, so ebx now effectively contains the value of the instruction pointer.
The Global Offset Table (GOT)
With this at hand, we can finally get to the implementation of position-independent data addressing on x86. It is accomplished by means of a “global offset table”, or in short GOT.

A GOT is simply a table of addresses, residing in the data section. Suppose some instruction in the code section wants to refer to a variable. Instead of referring to it directly by absolute address (which would require a relocation), it refers to an entry in the GOT. Since the GOT is in a known place in the data section, this reference is relative and known to the linker. The GOT entry, in turn, will contain the absolute address of the variable:

In pseudo-assembly, we replace an absolute addressing instruction:

; Place the value of the variable in edx
mov edx, [ADDR_OF_VAR]
With displacement addressing from a register, along with an extra indirection:

; 1. Somehow get the address of the GOT into ebx
lea ebx, ADDR_OF_GOT

; 2. Suppose ADDR_OF_VAR is stored at offset 0x10
; in the GOT. Then this will place ADDR_OF_VAR
; into edx.
mov edx, DWORD PTR [ebx + 0x10]

; 3. Finally, access the variable and place its
; value into edx.
mov edx, DWORD PTR [edx]
So, we’ve gotten rid of a relocation in the code section by redirecting variable references through the GOT. But we’ve also created a relocation in the data section. Why? Because the GOT still has to contain the absolute address of the variable for the scheme described above to work. So what have we gained?

A lot, it turns out. A relocation in the data section is much less problematic than one in the code section, for two reasons (which directly address the two main problems of load-time relocation of code described in the beginning of the article):

Relocations in the code section are required per variable reference, while in the GOT we only need to relocate once per variable. There are likely much more references to variables than variables, so this is more efficient.
The data section is writable and not shared between processes anyway, so adding relocations to it does no harm. Moving relocations from the code section, however, allows to make it read-only and share it between processes.
PIC with data references through GOT – an example
I will now show a complete example that demonstrates the mechanics of PIC:

int myglob = 42;

int ml_func(int a, int b)
{
return myglob + a + b;
}
This chunk of code will be compiled into a shared library (using the -fpic and -shared flags as appropriate) named libmlpic_dataonly.so.

Let’s take a look at its disassembly, focusing on the ml_func function:

0000043c :
43c: 55 push ebp
43d: 89 e5 mov ebp,esp
43f: e8 16 00 00 00 call 45a
444: 81 c1 b0 1b 00 00 add ecx,0x1bb0
44a: 8b 81 f0 ff ff ff mov eax,DWORD PTR [ecx-0x10]
450: 8b 00 mov eax,DWORD PTR [eax]
452: 03 45 08 add eax,DWORD PTR [ebp+0x8]
455: 03 45 0c add eax,DWORD PTR [ebp+0xc]
458: 5d pop ebp
459: c3 ret

0000045a :
45a: 8b 0c 24 mov ecx,DWORD PTR [esp]
45d: c3 ret
I’m going to refer to instructions by their addresses (the left-most number in the disassembly). This address is the offset from the load address of the shared library.

At 43f, the address of the next instruction is placed into ecx, by means of the technique described in the “key insight #2” section above.
At 444, a known constant offset from the instruction to the place where the GOT is located is added to ecx. So ecx now serves as a base pointer to GOT.
At 44a, a value is taken from [ecx – 0x10], which is a GOT entry, and placed into eax. This is the address of myglob.
At 450 the indirection is done, and the value of myglob is placed into eax.
Later the parameters a and b are added to myglob and the value is returned (by keeping it in eax).
We can also query the shared library with readelf -S to see where the GOT section was placed:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al

[19] .got PROGBITS 00001fe4 000fe4 000010 04 WA 0 0 4
[20] .got.plt PROGBITS 00001ff4 000ff4 000014 04 WA 0 0 4

Let’s do some math to check the computation done by the compiler to find myglob. As I mentioned above, the call to __i686.get_pc_thunk.cx places the address of the next instruction into ecx. That address is 0x444 [2]. The next instruction then adds 0x1bb0 to it, and the result in ecx is going to be 0x1ff4. Finally, to actually obtain the GOT entry holding the address of myglob, displacement addressing is used – [ecx – 0x10], so the entry is at 0x1fe4, which is the first entry in the GOT according to the section header.

Why there’s another section whose name starts with .got will be explained later in the article [3]. Note that the compiler chooses to point ecx to after the GOT and then use negative offsets to obtain entries. This is fine, as long as the math works out. And so far it does.

There’s something we’re still missing, however. How does the address of myglob actually get into the GOT slot at 0x1fe4? Recall that I mentioned a relocation, so let’s find it:

> readelf -r libmlpic_dataonly.so

Relocation section ‘.rel.dyn’ at offset 0x2dc contains 5 entries:
Offset Info Type Sym.Value Sym. Name
00002008 00000008 R_386_RELATIVE
00001fe4 00000406 R_386_GLOB_DAT 0000200c myglob

Note the relocation section for myglob, pointing to address 0x1fe4, as expected. The relocation is of type R_386_GLOB_DAT, which simply tells the dynamic loader – “put the actual value of the symbol (i.e. its address) into that offset”. So everything works out nicely. All that’s left is to check how it actually looks when the library is loaded. We can do this by writing a simple “driver” executable that links to libmlpic_dataonly.so and calls ml_func, and then running it through GDB.

> gdb driver
[…] skipping output
(gdb) set environment LD_LIBRARY_PATH=.
(gdb) break ml_func
[…]
(gdb) run
Starting program: […]pic_tests/driver

Breakpoint 1, ml_func (a=1, b=1) at ml_reloc_dataonly.c:5
5 return myglob + a + b;
(gdb) set disassembly-flavor intel
(gdb) disas ml_func
Dump of assembler code for function ml_func:
0x0013143c : push ebp
0x0013143d : mov ebp,esp
0x0013143f : call 0x13145a
0x00131444 : add ecx,0x1bb0
=> 0x0013144a : mov eax,DWORD PTR [ecx-0x10]
0x00131450 : mov eax,DWORD PTR [eax]
0x00131452 : add eax,DWORD PTR [ebp+0x8]
0x00131455 : add eax,DWORD PTR [ebp+0xc]
0x00131458 : pop ebp
0x00131459 : ret
End of assembler dump.
(gdb) i registers
eax 0x1 1
ecx 0x132ff4 1257460
[…] skipping output
The debugger has entered ml_func, and stopped at IP 0x0013144a [4]. We see that ecx holds the value 0x132ff4 (which is the address of the instruction plus 0x1bb0, as explained before). Note that at this point, at runtime, these are absolute addresses – the shared library has already been loaded into the address space of the process.

So, the GOT entry for myglob is at [ecx – 0x10]. Let’s check what’s there:

(gdb) x 0x132fe4
0x132fe4: 0x0013300c
So, we’d expect 0x0013300c to be the address of myglob. Let’s verify:

(gdb) p &myglob
$1 = (int *) 0x13300c
Indeed, it is!

Function calls in PIC
Alright, so this is how data addressing works in position independent code. But what about function calls? Theoretically, the exact same approach could work for function calls as well. Instead of call actually containing the address of the function to call, let it contain the address of a known GOT entry, and fill in that entry during loading.

But this is not how function calls work in PIC. What actually happens is a bit more complicated. Before I explain how it’s done, a few words about the motivation for such a mechanism.

The lazy binding optimization
When a shared library refers to some function, the real address of that function is not known until load time. Resolving this address is called binding, and it’s something the dynamic loader does when it loads the shared library into the process’s memory space. This binding process is non-trivial, since the loader has to actually look up the function symbol in special tables [5].

So, resolving each function takes time. Not a lot of time, but it adds up since the amount of functions in libraries is typically much larger than the amount of global variables. Moreover, most of these resolutions are done in vain, because in a typical run of a program only a fraction of functions actually get called (think about various functions handling error and special conditions, which typically don’t get called at all).

So, to speed up this process, a clever lazy binding scheme was devised. “Lazy” is a generic name for a family of optimizations in computer programming, where work is delayed until the last moment when it’s actually needed, with the intention of avoiding doing this work if its results are never required during a specific run of a program. Good examples of laziness are copy-on-write and lazy evaluation.

This lazy binding scheme is attained by adding yet another level of indirection – the PLT.

The Procedure Linkage Table (PLT)
The PLT is part of the executable text section, consisting of a set of entries (one for each external function the shared library calls). Each PLT entry is a short chunk of executable code. Instead of calling the function directly, the code calls an entry in the PLT, which then takes care to call the actual function. This arrangement is sometimes called a “trampoline”. Each PLT entry also has a corresponding entry in the GOT which contains the actual offset to the function, but only when the dynamic loader resolves it. I know this is confusing, but hopefully it will be come clearer once I explain the details in the next few paragraphs and diagrams.

As the previous section mentioned, PLTs allow lazy resolution of functions. When the shared library is first loaded, the function calls have not been resolved yet:

Explanation:

In the code, a function func is called. The compiler translates it to a call to func@plt, which is some N-th entry in the PLT.
The PLT consists of a special first entry, followed by a bunch of identically structured entries, one for each function needing resolution.
Each PLT entry but the first consists of these parts:
A jump to a location which is specified in a corresponding GOT entry
Preparation of arguments for a “resolver” routine
Call to the resolver routine, which resides in the first entry of the PLT
The first PLT entry is a call to a resolver routine, which is located in the dynamic loader itself [6]. This routine resolves the actual address of the function. More on its action a bit later.
Before the function’s actual address has been resolved, the Nth GOT entry just points to after the jump. This is why this arrow in the diagram is colored differently – it’s not an actual jump, just a pointer.
What happens when func is called for the first time is this:

PLT[n] is called and jumps to the address pointed to in GOT[n].
This address points into PLT[n] itself, to the preparation of arguments for the resolver.
The resolver is then called.
The resolver performs resolution of the actual address of func, places its actual address into GOT[n] and calls func.
After the first call, the diagram looks a bit differently:

Note that GOT[n] now points to the actual func [7] instead of back into the PLT. So, when func is called again:

PLT[n] is called and jumps to the address pointed to in GOT[n].
GOT[n] points to func, so this just transfers control to func.
In other words, now func is being actually called, without going through the resolver, at the cost of one additional jump. That’s all there is to it, really. This mechanism allows lazy resolution of functions, and no resolution at all for functions that aren’t actually called.

It also leaves the code/text section of the library completely position independent, since the only place where an absolute address is used is the GOT, which resides in the data section and will be relocated by the dynamic loader. Even the PLT itself is PIC, so it can live in the read-only text section.

I didn’t get into much details regarding the resolver, but it’s really not important for our purpose here. The resolver is simply a chunk of low-level code in the loader that does symbol resolution. The arguments prepared for it in each PLT entry, along with a suitable relocation entry, help it know about the symbol that needs resolution and about the GOT entry to update.

PIC with function calls through PLT and GOT – an example
Once again, to fortify the hard-learned theory with a practical demonstration, here’s a complete example showing function call resolution using the mechanism described above. I’ll be moving forward a bit faster this time.

Here’s the code for the shared library:

int myglob = 42;

int ml_util_func(int a)
{
return a + 1;
}

int ml_func(int a, int b)
{
int c = b + ml_util_func(a);
myglob += c;
return b + myglob;
}
This code will be compiled into libmlpic.so, and the focus is going to be on the call to ml_util_func from ml_func. Let’s first disassemble ml_func:

00000477 :
477: 55 push ebp
478: 89 e5 mov ebp,esp
47a: 53 push ebx
47b: 83 ec 24 sub esp,0x24
47e: e8 e4 ff ff ff call 467
483: 81 c3 71 1b 00 00 add ebx,0x1b71
489: 8b 45 08 mov eax,DWORD PTR [ebp+0x8]
48c: 89 04 24 mov DWORD PTR [esp],eax
48f: e8 0c ff ff ff call 3a0

The interesting part is the call to ml_util_func@plt. Note also that the address of GOT is in ebx. Here’s what ml_util_func@plt looks like (it’s in an executable section called .plt):

000003a0 :
3a0: ff a3 14 00 00 00 jmp DWORD PTR [ebx+0x14]
3a6: 68 10 00 00 00 push 0x10
3ab: e9 c0 ff ff ff jmp 370
Recall that each PLT entry consists of three parts:

A jump to an address specified in GOT (this is the jump to [ebx+0x14])
Preparation of arguments for the resolver
Call to the resolver
The resolver (PLT entry 0) resides at address 0x370, but it’s of no interest to us here. What’s more interesting is to see what the GOT contains. For that, we first have to do some math.

The “get IP” trick in ml_func was done on address 0x483, to which 0x1b71 is added. So the base of the GOT is at 0x1ff4. We can take a peek at the GOT contents with readelf [8]:

> readelf -x .got.plt libmlpic.so

Hex dump of section ‘.got.plt’:
0x00001ff4 241f0000 00000000 00000000 86030000 $……………
0x00002004 96030000 a6030000 ……..
The GOT entry ml_util_func@plt looks at is at offset +0x14, or 0x2008. From above, the word at that location is 0x3a6, which is the address of the push instruction in ml_util_func@plt.

To help the dynamic loader do its job, a relocation entry is also added and specifies which place in the GOT to relocate for ml_util_func:

> readelf -r libmlpic.so
[…] snip output

Relocation section ‘.rel.plt’ at offset 0x328 contains 3 entries:
Offset Info Type Sym.Value Sym. Name
00002000 00000107 R_386_JUMP_SLOT 00000000 __cxa_finalize
00002004 00000207 R_386_JUMP_SLOT 00000000 __gmon_start__
00002008 00000707 R_386_JUMP_SLOT 0000046c ml_util_func
The last line means that the dynamic loader should place the value (address) of symbol ml_util_func into 0x2008 (which, recall, is the GOT entry for this function).

It would be interesting to see this GOT entry modification actually happen after the first call. Let’s once again use GDB for the inspection.

> gdb driver
[…] skipping output
(gdb) set environment LD_LIBRARY_PATH=.
(gdb) break ml_func
Breakpoint 1 at 0x80483c0
(gdb) run
Starting program: /pic_tests/driver

Breakpoint 1, ml_func (a=1, b=1) at ml_main.c:10
10 int c = b + ml_util_func(a);
(gdb)
We’re now before the first call to ml_util_func. Recall that GOT is pointed to by ebx in this code. Let’s see what’s in it:

(gdb) i registers ebx
ebx 0x132ff4
And the offset to the entry we need is at [ebx+0x14]:

(gdb) x/w 0x133008
0x133008: 0x001313a6
Yep, the 0x3a6 ending, looks right. Now, let’s step until after the call to ml_util_func and check again:

(gdb) step
ml_util_func (a=1) at ml_main.c:5
5 return a + 1;
(gdb) x/w 0x133008
0x133008: 0x0013146c
The value at 0x133008 was changed. Hence, 0x0013146c should be the real address of ml_util_func, placed in there by the dynamic loader:

(gdb) p &ml_util_func
$1 = (int (*)(int)) 0x13146c
Just as expected.

Controlling if and when the resolution is done by the loader
This would be a good place to mention that the process of lazy symbol resolution performed by the dynamic loader can be configured with some environment variables (and corresponding flags to ld when linking the shared library). This is sometimes useful for special performance requirements or debugging.

The LD_BIND_NOW env var, when defined, tells the dynamic loader to always perform the resolution for all symbols at start-up time, and not lazily. You can easily verify this in action by setting this env var and re-running the previous sample with GDB. You’ll see that the GOT entry for ml_util_func contains its real address even before the first call to the function.

Conversely, the LD_BIND_NOT env var tells the dynamic loader not to update the GOT entry at all. Each call to an external function will then go through the dynamic loader and be resolved anew.

The dynamic loader is configurable by other flags as well. I encourage you to go over man ld.so – it contains some interesting information.

The costs of PIC
This article started by stating the problems of load-time relocation and how the PIC approach fixes them. But PIC is also not without problems. One immediately apparent cost is the extra indirection required for all external references to data and code in PIC. That’s an extra memory load for each reference to a global variable, and for each call to a function. How problematic this is in practice depends on the compiler, the CPU architecture and the particular application.

Another, less apparent cost, is the increased register usage required to implement PIC. In order to avoid locating the GOT too frequently, it makes sense for the compiler to generate code that keeps its address in a register (usually ebx). But that ties down a whole register just for the sake of GOT. While not a big problem for RISC architectures that tend to have a lot of general purposes registers, it presents a performance problem for architectures like x86, which has a small amount of registers. PIC means having one general purpose register less, which adds up indirect costs since now more memory references have to be made.

Conclusion
This article explained what position independent code is, and how it helps create shared libraries with shareable read-only text sections. There are some tradeoffs when choosing between PIC and its alternative (load-time relocation), and the eventual outcome really depends on a lot of factors, like the CPU architecture on which the program is going to run.

That said, PIC is becoming more and more popular. Some non-Intel architectures like SPARC64 force PIC-only code for shared libraries, and many others (for example, ARM) include IP-relative addressing modes to make PIC more efficient. Both are true for the successor of x86, the x64 architecture. I will discuss PIC on x64 in a future article.

The focus of this article, however, has not been on performance considerations or architectural decisions. My aim was to explain, given that PIC is used, how it works. If the explanation wasn’t clear enough – please let me know in the comments and I will try to provide more information.

[1] Unless all applications load this library into the exact same virtual memory address. But this usually isn’t done on Linux.
[2] 0x444 (and all other addresses mentioned in this computation) is relative to the load address of the shared library, which is unknown until an executable actually loads it at runtime. Note how it doesn’t matter in the code since it only juggles relative addresses.
[3] The astute reader may wonder why .got is a separate section at all. Didn’t I just show in the diagrams that it’s located in the data section? In practice, it is. I don’t want to get into the distinction between ELF sections and segments here, since that would take use too far away from the point. But briefly, any number of “data” sections can be defined for a library and mapped into a read-write segment. This doesn’t really matter, as long as the ELF file is organized correctly. Separating the data segment into different logical sections provides modularity and makes the linker’s job easier.
[4] Note that gdb skipped the part where ecx is assigned. That’s because it’s kind-of considered to be part of the function’s prolog (the real reason is in the way gcc structures its debug info, of course). Several references to global data and functions are made inside a function, and a register pointing to GOT can serve all of them.
[5] Shared library ELF objects actually come with special hash table sections for this purpose.
[6] The dynamic loader on Linux is just another shared library which gets loaded into the address space of all running processes.
[7] I placed func in a separate code section, although in theory this could be the same one where the call to func is made (i.e. in the same shared library). The “extra credit” section of this article has information about why a call to an external function in the same shared library needs PIC (or relocation) as well.
[8] Recall that in the data reference example I promised to explain why there are apparently two GOT sections in the object: .got and .got.plt. Now it should become obvious that this is just to conveniently split the GOT entries required for global data from GOT entries required for the PLT. This is also why when the GOT offset is computed in functions, it points to .got.plt, which comes right after .got. This way, negative offsets lead us to .got, while positive offsets lead us to .got.plt. While convenient, such an arrangement is by no means compulsory. Both parts could be placed into a single .got section.
Comments

© 2003-2018 Eli Bendersky Back to top

Fedora Project: Changes/Harden All Packages

Fedora Project: Changes/Harden All Packages

Harden All Packages
Summary

In Fedora 22 and before, it was up to the package maintainer to add %global _hardened_build 1 to their spec file to ensure their program was hardened. Beginning with Fedora 23 this will now become the defaults for all packages. You can compare the security by running the following as root:

dnf install checksec
checksec –proc-all
To see the mitigations in the libraries that a process is using, find the process id from the previous command (for example 123):

checksec –proc-libs 123
Owner

Name: Till Maas | Moez Roy | Adam Jackson | Florian Weimer
Email: opensource@till.name | moez.roy@gmail.com | ajax@redhat.com | fweimer@redhat.com
FESCo shepherd: Adam Jackson

Current status

Targeted release: Fedora 22 Fedora 23
Last updated: 2015-02-12
Tracking bug: #1215939
Build failures tracking bug: #1199775
Detailed Description

Currently, the Packaging Guidelines allow maintainers to decide whether their packages use position-independent code (PIC). There are rules that say that a lot of packages should use PIC, but in reality a lot of packages do not use PIC even if they must. Also since a lot of packages if not all potentially process untrusted input, it makes sense for these packages to use PIC to enhance the security of Fedora. Therefore I propose to build all packages with PIC by changing RPM to use the appropriate flags by default.

Implementation (done): Change line 130 in redhat-rpm-config macros from #_hardened_build 0 to %_hardened_build 1

References:

https://fedorahosted.org/fesco/ticket/1384
https://fedorahosted.org/fesco/ticket/1113 (older attempt with many references)
https://fedorahosted.org/rel-eng/ticket/6049
There should be several mails about this on the devel list
Build failures tracker bug
Detailed Harden Flags Description
The following only deals with flags added by this proposal. It does not mention existing hardening flags such as the -fstack-protector-strong compiler flag or the -z relro linker flag.

This table shows the required compiler and linker flags with and without full ASLR support. The flags are described from the perspective of the gcc or g++ compiler driver, so linker flags (for ld) are prefixed with -Wl.

Non-hardening Hardening
Compile for static linking (nothing) -fPIE
Link executable (nothing) -pie -Wl,-z,now
Compile for dynamic linking -fPIC -fPIC
Link shared object -shared -shared -Wl,-z,now

The key change is that for PIE builds, compilation for static linking (such as object files which go into the main program, not a library) needs the flag -fPIE. But this flag must not be included when compiling for dynamic linking because the resulting object code is not compatible with that. To repeat, you should not specify both -fpic and -fpie on the same command line because this rarely has the intended effect.

For both hardened and non-hardened builds, it is possible to compile everything with -fPIC -pie. This can simplify injecting the flags into the build process.

Alternatively, you can rely on the RPM-provided compiler and linker flag settings. In Fedora 23, this will enable additional GCC specs files (which are not related to RPM spec files), altering the compiler driver behavior in the following way:

If there is no -fPIC-style flag on the command line, the source file is compiled as if -fPIE were specified.
If there is no -shared flag on the command line, the program is linked with -pie.
-z now is always passed to the linker.
This happens by injection CFLAGS, CXXFLAGS, and LDFLAGS environment variables in the invocation of the %configure RPM macro. Details are in the /usr/lib/rpm/redhat/macros file. (In previous Fedora releases, this only happened when the _hardended_build RPM macro was set to 1 in the spec file.)

In effect, this maps the non-hardening column to the hardening column in the table above.

Copy relocations support in GCC 5 and binutils 2.26 makes the performance on x86_64 of PIE literally zero for many programs.

redhat-hardened-cc1

*cc1_options:
+ %{!fpie:%{!fPIE:%{!fpic:%{!fPIC:%{!fno-pic:-fPIE}}}}}
redhat-hardened-ld

*self_spec:
+ %{!shared:-pie}

*link:
+ -z now
How To Test

Running checksec should always report only
Full RELRO Canary found NX enabled PIE enabled No RPATH No RUNPATH otherwise a tracking bug should exist for the respective packages. All packages need to be tested as usual for normal behaviour.

Troubleshooting steps for package maintainers
1. Add %global _hardened_build 1 and build your package against F21. As F21 uses GCC 4.9.2, older binutils, older glibc etc. you will be able to identify whether the build failure is caused by GCC5, the newer binutils, the new glibc etc..

2. If you get linker errors, you can try disabling -z now by putting the following under the %build section of your spec file:

CFLAGS=”$RPM_OPT_FLAGS -Wl,-z,lazy”
CXXFLAGS=”$RPM_OPT_FLAGS -Wl,-z,lazy”

export CFLAGS
export CXXFLAGS
3a. Check whether it builds successfully if you disable the hardening change:

%undefine _hardened_build
3b. Enable only PIE manually for your package:

CFLAGS=”$RPM_OPT_FLAGS -fPIC -pie”
CXXFLAGS=”$RPM_OPT_FLAGS -fPIC -pie”

export CFLAGS
export CXXFLAGS
3c. Enable only -z now manually for your package:

CFLAGS=”$RPM_OPT_FLAGS -Wl,-z,relro -Wl,-z,now”
CXXFLAGS=”$RPM_OPT_FLAGS -Wl,-z,relro -Wl,-z,now”

export CFLAGS
export CXXFLAGS
3d. Enable 3b and 3c above manually for your package:

CFLAGS=”$RPM_OPT_FLAGS -fPIC -pie -Wl,-z,relro -Wl,-z,now”
CXXFLAGS=”$RPM_OPT_FLAGS -fPIC -pie -Wl,-z,relro -Wl,-z,now”

export CFLAGS
export CXXFLAGS

Benefit to Fedora

Packages in Fedora will be more secure than in other distributions or packages provided by upstream. Therefore our users less likely become victims of attacks. Fedora will use more state-of-the-art security mechanisms to fulfill its first and features foundations.

Scope

Proposal owners:
Help writing the new packaging guidelines.

Other developers:
Change the rpm macros to build packages by default with PIC/PIE flags (i.e. set _hardened_package to 1 by default). Bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1192183

Release engineering:
Do a mass rebuild for all arch packages

Policies and guidelines:
Adjust the Packaging Guidelines to allow non-PIC packages only if the package is not working otherwise and require a tracker bug similar to packages not working on certain archs. Update the Guidelines to reflect the new defaults.

Upgrade/compatibility impact

This should not cause problems with upgrades.

User Experience

Fedora users might notice less sucessful attacks on their systems.

Dependencies

The rpm macros for Fedora need to be adjusted. Prelink might be retired.

Contingency Plan

Contingency mechanism: Package maintainers should refer to the section Troubleshooting steps for package maintainers above if they don’t want their FTBFS packages retired.
Contingency deadline: beta freeze
Blocks release? No
Documentation

The current packaging guidelines can be consulted.